Use a bound VACUUM INTO path and remove the stale SQLite
handle declaration. Fix shell continuations in Windows CI and
Debian installation instructions.
Refs discussion #886: a saved-report manager built on custom SQL and
nomenclature.json. Most of what was asked for already existed --
Projet > Exporter au format CSV already builds/saves/reuses named
SELECT queries via ElementQueryWidget and nomenclature.json, and
Projet > Ajouter une nomenclature already inserts one into a folio.
This fills the three real gaps.
- projectDataBase::isReadOnlySelect() rejects anything that isn't a
single SELECT/WITH statement. Checked in newQuery() itself, the one
choke point every query path already goes through -- including a
query loaded from a saved nomenclature/summary table's <query>
element on project open, not just the dialog's own custom-SQL box.
ElementQueryWidget shows the same check live as you type, and
BOMExportDialog surfaces it before running or exporting anything.
- BOMExportDialog gains a Preview button + table (QSqlQueryModel),
so a report can be checked on screen before committing to a CSV file.
- ElementQueryWidget gains Importer.../Exporter... buttons that
read/write nomenclature.json's saved reports as a JSON file, so a
report can be handed to a colleague or another install. Import asks
before overwriting a locally-saved report of the same name.
Verified: Qt 6.10.2, builds clean, ctest 8/8. Drove the real dialog
through Xvfb: typed "DROP TABLE element" into the custom-SQL box and
got the inline warning immediately, then confirmed Preview also
refuses it with a "Requête refusée" dialog rather than running it.
Preview against the real default query returned live column headers
and a row. Export opens a save dialog without crashing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
A slave and a terminal are both routinely separately orderable hardware. A
circuit breaker can carry ten or twenty auxiliary blocks, each with its own
order code, and a terminal block is a purchased part in its own right.
Neither was reaching the bill of materials.
Decided in discussion #847: @IBSYSLevi -- "I would not expect that a defined
piece of hardware is excluded from BOM when not specifically defined as so" --
with use cases from @jozi332 covering Siemens breakers with ten to twenty
auxiliary blocks and PLC cards carrying per-channel data.
Two filters had to change, which is easy to miss: BomExport::defaultQuery()
and, upstream of it, the WHERE clause of element_nomenclature_view itself.
Changing only the query does nothing for slaves, because the view had already
removed them. Terminals were already in the view, so they appeared as soon as
the query allowed them -- which made a half-finished change look like it had
worked.
Measured on examples/industrial.qet, which holds 96 terminals and 41 slaves:
258 rows before, 354 with terminals, 395 with both. A slave given a
manufacturer and part number now appears in the export; previously it could
not, at any setting.
Nothing that should stay out of a bill of materials is newly included. The
folio report arrows and the conductor definition are still excluded because
they are not hardware, and anything else -- a relay's own auxiliary contact,
which is not orderable separately -- is kept out with exclude_from_bom, which
the view already honours and which #721 and #765 made settable on the symbol
itself.
tst_smart_device is updated rather than weakened. @enesgursoy6110 wrote it in
#830 to prove the filter works, inserting rows designated "Must not be
exported"; the slave and terminal rows now carry real designations and are
asserted present, and a folio report arrow takes over as the negative case,
so the test still proves filtering happens -- at the boundary we now want.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
ProjectDBModel::setQuery() calls projectDataBase::updateDB(), which drops
and repopulates every table in the database. The rebuild does not depend on
the query, so each table model that queries the database while a project is
being read triggers another complete repopulate of the same content.
Opening a 100 folio project ran updateDB() 26 times, 9.9 s of a 15.7 s load.
Two changes, because the first alone is not enough:
setUpdateBlocked() lets a bulk operation suppress the rebuild and do it
once when it is done. readProjectXml() already wrapped the load in
blockSignals(true) "to avoid hundreds of unnecessary emitted signal", but
that suppresses only the signal, not the work it announces; this extends
the same intent to the work. Both early returns in readProjectXml() sit
before the block, so no path leaves the database permanently blocked.
Further rebuilds are triggered after readProjectXml() returns, where the
load phase timers cannot see them -- with only the block in place
updateDB() still ran 5 times on examples/industrial.qet. So the database
now also tracks whether anything has changed since the last rebuild, and
skips repopulating when nothing has. dataBaseUpdated() is still emitted in
that case: callers and models rely on it to refresh, and what they read
back is the same either way. Every method of the class that writes rows
marks the flag; from outside, the database is reachable only through
newQuery(), and all five call sites read.
Repeating the rebuild was wasteful rather than wrong -- each
populate*Table() begins with a DELETE -- so this changes no output.
Verified byte identical --info on all 23 example projects, and identical
--export-bom, --export-wires, --export-cables, --export-nets and
--export-wiring on industrial.qet. Its load drops from 5.51 s to 5.31 s
(median of 6).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L6MRq2Ach1ogvnGcbuqNLr
Each cascade delete (element_info/terminal/conductor/element) only
logged its error and fell through to the next query regardless, so a
mid-cascade failure (e.g. a locked DB) still let the diagram row get
deleted while its child rows survived -- the same inconsistency this
branch set out to fix, just via a different failure path. Wrap the
cascade in a transaction and roll back + bail on the first failed
exec(), matching the existing transaction pattern in updateDB().
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
removeDiagram() only ever deleted the diagram's own row. No foreign key in
this schema is declared ON DELETE CASCADE (and SQLite foreign-key
enforcement is never turned on for this connection anyway), so removing a
diagram left every element, element_info, terminal and conductor row that
belonged to it behind in the database -- silently, since nothing reads them
until the next full updateDB() rebuild papers over it.
Traced why this had never crashed anything: Diagram::~Diagram() explicitly
walks and deletes its top-level items through removeItem() (which does call
dataBase()->removeElement() correctly), but deliberately skips conductors --
because a conductor's destructor touches both of its terminals
(terminal1->removeConductor(this)), and those terminals may belong to an
element already destroyed earlier in the same sweep. Conductors are instead
destroyed as a side effect of Terminal::~Terminal()'s qDeleteAll() on its own
conductor list, which is a plain C++ delete that never goes through
Diagram::removeItem() and therefore never calls dataBase()->removeConductor()
at all. So the object graph is torn down safely, but the database is never
told about the conductors or their terminals.
Fixed by adding the missing bulk deletes to projectDataBase::removeDiagram()
itself, run while the diagram (and its live scene) still exist -- verified
that QETProject::detachDiagram() emits diagramRemoved() (which this class's
constructor connects to this slot) synchronously, before the Diagram object
is scheduled for destruction via deleteLater(), so nothing here races the
C++ teardown described above. Order matters: element_info and terminal have
no diagram_uuid column of their own, so both are scoped through a subquery
on element and must run before element itself is deleted.
Verified against examples/industrial.qet (50 diagrams) by calling
projectDataBase::removeDiagram() directly and comparing table counts before
and after, with no intervening updateDB() call to mask a gap:
element=354->335 element_info=354->335 terminal=1087->1033 conductor=671->626 diagram=50->49
Every delta matches a direct SQL count for that diagram's own rows exactly
(19 elements, 54 terminals), and both "orphan rows still referencing the
removed diagram" checks read 0 afterward -- so the cascade is complete and,
just as importantly, scoped: nothing belonging to the other 49 diagrams
moved.
Separate finding, not fixed here: QETProject::removeDiagram(Diagram*) (the
synchronous, non-undoable variant, not the usual GUI
ProjectView::removeDiagram() path) segfaults if the enclosing QETProject is
destroyed before an event loop iteration lets its pending deleteLater() run
-- reproduces identically on unmodified master, so it predates and is
unrelated to this change. Worth its own report; a headless caller is the
only realistic way to hit it, which is how this surfaced.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
excludedConductorCount() counted conductors whose terminals had no uuid,
which was the right rule when that was the reason they were dropped. It no
longer is: Terminal::stableUuid() derives an identity from the terminal's
geometry, so those conductors are in the table.
Left unchanged, the dialog would have told the user "671 conductors excluded"
on industrial.qet while listing all 671 of them -- a worse failure than the
one the count exists to prevent, because it undermines a list that is now
correct.
The count and the dialog's explanation both now describe the case that
actually remains: an endpoint attached to no element at all, which has no
identity to key on under any scheme.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The element and element_info tables had two independent insert paths --
addElement() for an element added to a live diagram, and
populateElementTable()/populateElementInfoTable() for a full rebuild --
which bound the same row differently. The incremental path wrote
kindInformations()["type"] into element.sub_type; the bulk path wrote
elementData().masterTypeToString(). So the table held different values
depending on whether the project had been reloaded since the element was
placed, and element_nomenclature_view exposes that column as
element_sub_type, which ElementQueryWidget filters on for the Coil,
Protection, Commutator and PLC nomenclature options.
That divergence is the same shape as the type-filter one fixed in the
previous commit, and it is the reason this stack kept finding bugs that
were invisible while editing and only appeared after a reload. Rather
than correct a second instance of it, both paths now go through
bindElementValues() and bindElementInfoValues(), following the
bindDiagramInfoValues() helper this class already had. Live and reloaded
now agree by construction instead of by coincidence.
The bulk path's values are the ones kept, because they are what every
already saved project contains: nothing a reload produces changes, and
the previous commit's 19-project BOM regression stays valid. It is the
live path that moves, onto the values a reload would have given it
anyway.
Measured, placing one element into a new project and then saving and
reopening it:
live element table: slave/ x1
reloaded element table: slave/ x1
and for the same element, what the two paths would have stored:
bulk (now shared): "" incremental (before this commit): "simple"
Re-ran the BOM regression over the same 19 projects after this change:
content identical to the pre-change baseline on all 19, and identical
line-for-line on 18, the exception being the three byte-identical
photovoltaique rows already described in the previous commit.
Note for anyone reading masterTypeToString(): the const no-argument
overload returns an empty string for anything that is not a Master, so
the "coil" fallback in the static overload is only reached for real
master elements. Non-master elements get an empty sub_type, not a
spurious "coil".
Closes the gap left open by the previous commit, at the root rather than
around it.
populateElementTable()/populateElementInfoTable() only inserted elements
matching Simple|Terminal|Master|Thumbnail. That quietly made the element
table mean "the elements a nomenclature cares about" rather than "the
elements of the project": slave elements (relay contacts) and report
elements -- ordinary conductor endpoints -- had no row at all after a
project load, so the wiring list could not name either end of a wire
that terminated on one.
Both tables are now populated with every ElementData::Type, and the type
restriction moves into element_nomenclature_view, which is where a
"what belongs in a bill of materials" decision belongs. The mask in the
view is character-for-character the one the population used to apply, so
a relay contact is still not a BOM line item.
This is safe to do in one place because every consumer of the project
database goes through a view: element_nomenclature_view (the on-diagram
nomenclature table via ElementQueryWidget, the BOM dialog, and the
--export-bom CLI) or project_summary_view (which does not reference
element at all). Nothing queries the element or element_info tables
directly -- checked across the whole tree.
Regression evidence. --export-bom runs updateDB() and then queries
element_nomenclature_view, so it is an exact harness for what the GUI
BOM shows. Captured for 19 projects (all 17 usable examples/ plus two
slave-element fixtures) before and after:
- BOM content byte-identical on all 19, compared as a multiset.
- 18 of 19 are also identical line-for-line in order.
- photovoltaique differs only in the position of three byte-identical
rows among themselves. Its query is ORDER BY label and those rows
share an empty label, so their relative order was never defined;
they are indistinguishable in the output. The on-diagram
nomenclature orders by every displayed column, so a tie there means
the rows are identical on screen too.
Effect on the wiring list, same project and same reload path: element_info
rows 0 -> 2, and the two component columns go from blank to K2 -> K1.
Cost: the database phase of loading examples/industrial.qet (150 folios,
1794 terminals) moves from 0.210 s to 0.233 s.
Slice 4 of discussion #503, on top of slice 3 (#629): the smallest
surface that makes wiring_list_view visible, plus the diagnostic the
view needs to be honest about what it is missing.
Projet > "Liste de câblage (base de données)" opens a read-only table of
wiring_list_view, headed by a line stating how many conductors are
listed and, when non-zero, how many were excluded and why.
Deliberately not another exporter. QET already ships a wiring-list CSV
export (Projet > Exporter le plan de câblage, and --export-cables) which
walks the project XML; measured on the same projects it produces a row
per conductor and resolves labels correctly when the project has them.
Adding a second, competing CSV would be worse, not better -- the
database path's value is what it unlocks (terminal plans, BOM joins),
not replacing that export.
projectDataBase::excludedConductorCount() counts, from the live scene,
the conductors deliberately absent from the conductor table because a
terminal has no uuid. Counted from the scene precisely because the
database is where those conductors are not. Verified: 671 on
examples/industrial.qet (which has 1794 terminals and no terminal uuids
at all, so its list is empty and now says so), 0 on a project whose
elements do carry terminal uuids.
KNOWN GAP, not fixed here and the reason this is opened for discussion
rather than merge: after a save/reload the component columns are blank
for slave elements. populateElementTable()/populateElementInfoTable()
only insert Simple|Terminal|Master|Thumbnail, so slave elements -- relay
contacts, i.e. a large share of real wire endpoints -- have no row in
element_info for the view to read a label from. Measured on a two-slave-
contact project after reload: element rows 0, element_info rows 0,
terminal rows 2, conductor rows 1; the wire is listed (slice 3's LEFT
JOIN keeps it) but both component names are empty, where the existing
CSV export shows K1 -> K2 for the same file.
Closing that gap means widening a filter shared with the nomenclature
and summary views, which would change what those existing, shipped
features contain. That is a maintainer decision, not one to take
unilaterally inside an additive slice.
The comment above this view promised that it "returns exactly as many rows
as the conductor table holds", and argued carefully for the two joins that
could have broken that -- no inner join to element, and element_info LEFT
joined. Then it ended with an inner join to diagram that it never mentioned,
which can drop rows just as easily.
Feeding the real schema a conductor whose diagram_uuid has no diagram row
returned 2 view rows for 3 conductors. With the join made LEFT it returns 3,
with a null folio instead of a missing wire.
In practice this should never fire: QETProject::diagramAdded is connected to
addDiagram(), so the folio exists before anything can be drawn on it. But an
inner join turns that into an assumption the view enforces silently, and of
all the things this view can get wrong, dropping a wire from a wiring list
is the one that matters most. The comment now says which joins are inner and
why those two are safe.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Slice 3 of discussion #503, on top of slice 2 (#628). One row per
conductor, each endpoint resolved to its element label and terminal
name -- the `F1:4 -> M200:U1` shape from the original prototype.
The view deviates from the SQL sketched in the discussion in two ways,
both because the sketched version silently loses wires:
- **No join to the `element` table.** A terminal row already carries its
`element_uuid`, so joining `element` back just to read the same uuid
adds nothing. Worse, it filters: `populateElementTable()` only inserts
elements matching `Simple|Terminal|Master|Thumbnail`, so `Slave`
elements (relay contacts and the like -- extremely common at the end
of a wire) and report elements are simply absent from that table after
a project load, and an inner join through it drops their conductors.
- **`element_info` is LEFT joined** for the same reason. A wire whose
endpoint element has no info row still belongs in a wiring list; it
comes back with an empty label rather than vanishing. Losing a wire
from a wiring list is a worse failure than showing one with a blank
end.
Note this only bites after a save/reload. The incremental `addElement()`
path does not apply the type filter, so a slave element placed live is
present in `element`/`element_info` and an inner join looks fine -- it
is the bulk repopulate on project load that drops it. Testing only the
live-editing path would have missed this entirely.
Measured, comparing this view against an inner-join-through-element
variant built from the same tables in the same session:
| project | conductors | wiring_list_view | inner-join variant |
|---|---|---|---|
| Polonez MR'89 wiring diagram | 280 | 280 | 280 |
| two slave contacts, after save+reload | 1 | **1** | **0** |
Polonez happens to have no slave elements at conductor ends, so both
agree there and the problem is invisible. The second case is the
minimal reproduction: place two "Simple contact" elements
(`link_type="slave"`) so autoconnect wires them, save, reload -- the
sketched view returns zero rows for a project that plainly has a wire
in it.
Acceptance criterion held throughout: `wiring_list_view` row count
equals `conductor` row count, i.e. the view itself drops nothing.
Conductors already excluded upstream (legacy terminals without uuids,
see #628) stay excluded; that remains the only thing missing from the
list, and is what slice 4 should surface a count for.
The conductor table keyed on Terminal::uuid(), which comes from the catalog
.elmt definition and is empty for every element authored before that field
existed. A conductor was dropped unless *both* its terminals had one, so the
tables this slice adds were empty on almost every project in existence:
examples corpus conductor rows in the database
industrial.qet 0 of 671
affuteuse_250h.qet 0 of 263
tremie_vibrante.qet 0 of 77
741.qet 0 of 67
Across the 23 example projects, 16 of the 20 that contain conductors have
zero terminal uuids -- 2366 of 3002 conductors -- and overall coverage is
7.3%. Meanwhile --export-cables, already on master, lists all 671 conductors
of industrial.qet from the document. A feature that only works on newly
authored elements is not one users can rely on.
Terminal::stableUuid() returns the terminal's own uuid when it has one and
otherwise derives one from its local position and orientation inside its
element. That is not an invented scheme: it is what the project format
already does. TerminalData::fromXml() says so where it parses the field --
"if the attribute not exists, means, the element is created with an older
version of qet. So use the legacy approach to identify terminals" -- and the
legacy approach is the terminal's position. m_pos is read from the definition
and is not touched by moving the element on a folio, so the identity survives
loads, saves and folio moves. Derived values are UUID v5 in a fixed namespace,
so they are reproducible without being stored, and cannot collide with the v4
uuids the element editor generates.
Every project in the corpus now has exactly as many conductor rows as the
document has conductors -- 20 of 20 measured, 0 mismatches. (schema_indus.qet
is excluded: it blocks on a modal dialog at zero CPU under any CLI flag, the
pre-existing hang PR #661 addresses.)
Two things this deliberately does not key on:
- The terminal name. It is not stable: QET rewrites a terminal named "_" as
unnamed, which would have silently changed the identity of 1421 of
industrial.qet's 1790 terminals on their first resave. Measured across the
corpus, dropping it costs nothing -- geometry alone yields exactly the same
three collisions -- and it means renaming a terminal no longer changes what
it is.
- Uniqueness in the face of a definition that declares two terminals at the
same point and orientation. Three cases exist in the whole corpus. They
merge to a single terminal row, which is harmless: two terminals identical
in position and orientation are indistinguishable in every observable
respect, and every conductor on either still resolves to the right element
and terminal name. Both affected projects (industrial, perceuse) return
their full conductor count.
The only conductor still skipped is one whose terminal has no parent element,
which has no identity to key on at all.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Three fixes to the tables added by this slice.
A conductor's text was written once at insert and never again. Renaming a
wire left the database holding the old number, so the wiring list showed a
stale value until the next full repopulate -- elements have
elementInfoChanged() for exactly this and conductors had nothing.
Conductor::setProperties() has around a dozen call sites (auto-numbering,
the properties dialog, element moves, the delete command's re-links), so
rather than adding a call to each and missing the ones added later, listen
to the propertiesChange() signal it already emits. Qt::UniqueConnection
means a repeated insert or a full repopulate cannot double-subscribe, and
the connection is established on both insert paths because conductors read
from a file never pass through addConductor().
addConductor() and populateConductorTable() each carried their own copy of
the same seven bindValue() lines. They had not drifted yet, but that is the
same duplication the element paths had before bindElementValues(), where
they had drifted -- one binding kindInformations()["type"] and the other
masterTypeToString(). One bindConductorValues() for both.
Finally, index the conductor columns that get looked up per element rather
than per conductor. element_nomenclature_view counts the wires touching each
element with a correlated subquery, so without an index every element row
full-scans the conductor table and the cost grows as elements x conductors.
Measured on a standalone SQLite harness at 2000 elements x 5000 conductors:
2134 ms unindexed, 10 ms indexed. diagram_uuid is indexed too, since the
wiring list view joins on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Slice 2 of discussion #503 (from-to wiring list built on projectDataBase),
building on the conductor uuid from slice 1 (#625). Pure plumbing: two
new additive tables plus their populate/add/remove hooks. No view, no UI,
no visible behavior change yet -- the wiring-list view is slice 3.
Follows the existing shape of the class throughout: same table/column
naming, same prepared-statement idiom in prepareQuery(), same
bind/exec/qDebug-lastError error handling, same DELETE-then-loop
populate pattern.
- `terminal (uuid, element_uuid, name)` and
`conductor (uuid, diagram_uuid, terminal1_uuid, terminal1_element_uuid,
terminal2_uuid, terminal2_element_uuid, text)` created alongside the
existing tables in createDataBase().
- populateConductorTable() added as a fifth populate* call in updateDB().
Terminal population is folded into it, since a terminal only matters
here in the context of a conductor referencing it.
- addConductor()/removeConductor() hooked into the already-existing
Conductor::Type branch of Diagram::addItem()/removeItem(), mirroring
the Element::Type branch directly above.
Two things the original schema sketch in the discussion got wrong, found
by testing rather than inspection:
1. Terminal::uuid() is NOT unique per placed terminal. It is the
terminal-position id baked into the catalog .elmt definition ("the
top terminal"), so every placed instance of the same catalog element
shares it. A terminal instance is only uniquely identified by
(uuid, element_uuid) together, so that pair is the terminal table's
primary key and the conductor table carries both halves for each
endpoint. With uuid alone as PK, the second placed instance of any
element silently lost its terminals to the INSERT OR IGNORE.
2. Conductors whose terminals predate terminal uuids are omitted rather
than given a fabricated identity, as agreed in the discussion. This
turns out to matter far more than expected in practice -- see below.
Testing (all live, in the running app):
- Incremental add: fresh project, two vertically aligned contacts placed
so autoconnect creates a conductor -> 2 terminals, 1 conductor.
- Incremental remove: deleting that conductor -> conductor count 1 -> 0.
- Undo: ctrl+Z after the delete -> back to 1, no duplicate-primary-key
error (the same Conductor object keeps its uuid).
- Bulk populate: examples/weneedpolonez-Polonez_MR89_wiring_diagram.qet
(366 conductors) -> 478 terminals, 280 conductors; the 86 conductors
touching legacy terminals correctly omitted.
- Join correctness: conductor -> terminal (composite key) -> element_info
resolves real from-to rows with real element labels.
- Legacy-only project: examples/industrial.qet has 1794 terminals and
*zero* terminal uuids, so all 671 of its conductors are omitted. Loads
and renders fine, no crash, no spurious rows -- but worth stating
plainly that a from-to wiring list for that project would be empty
today. This is a property of the element catalog definitions, not of
the project file, and is the strongest argument for surfacing an
"N conductors excluded" count to the user when the view lands.
- No SQL errors logged in any of the above.
Known limitation, consistent with existing behavior: removeDiagram()
does not cascade-delete the conductor rows of that diagram, exactly as
it already does not cascade to element/element_info. A full updateDB()
rebuild clears them, and the future wiring-list view INNER JOINs from
conductor, so orphan terminal rows never surface.
QSqlDatabase::exec(const QString&) is deprecated in Qt6. Route the
PRAGMA/CREATE TABLE statements through QSqlQuery(db).exec() instead.
Clears 11 -Wdeprecated-declarations warnings; same statements, same
database connection, no behavioural change.
clazy is a compiler plugin which allows clang to understand Qt
semantics. You get more than 50 Qt related compiler warnings, ranging
from unneeded memory allocations to misusage of API, including fix-its
for automatic refactoring.
https://invent.kde.org/sdk/clazy
All export files that are derived from the project (BOM,
nomenclature, etc.) are saved in the same directory by default.
In this context, the standard directories have been grouped
together in qetapp.cpp / qetapp.h so that only one place needs
to be searched for in case of any adjustments.
Replace QETApp::diagramInfoKeys() by QETInformation::diagramInfoKeys()
and QETApp::diagramTranslatedInfoKey(str) by
QETInformation::translatedInfoKey(str)
Magnus Hellströmer