Reviving #664, closed 2026-09-18 purely to clear a maintainer review
backlog (#630), not on merit; not superseded. Rewritten fresh against
current master rather than merged from the old branch -- that branch
predates the Qt6-only switch and much of dataBase/projectdatabase.cpp's
later rewrite, and the two had diverged too far for a textual merge to
be trustworthy.
projectDataBase::removeElement() only ran DELETE FROM element WHERE
uuid=:uuid. It never touched element_info, even though every element
also has a row there (element_uuid is its PRIMARY KEY, with a FOREIGN
KEY back to element.uuid that isn't enforced by this connection -- no
ON DELETE CASCADE in effect). So deleting an element left its
element_info row orphaned.
Re-adding an element with that same uuid later -- undo of that same
deletion, or a redo replaying it -- goes through addElement(), which
INSERTs into both tables. The element insert succeeds (that row really
was removed). The element_info insert hits the orphaned row's primary
key and fails, silently: the error is logged and swallowed, so the
element re-enters the scene with no element_info row at all, and
nothing later re-syncs it.
removeDiagram() already cascades this cleanup when a whole folio is
removed (a later, unrelated addition) -- confirmed on current master --
but that path never runs for a single element removed on its own,
which is the case this fixes.
Verified on the built binary, not just read: placed an element, deleted
it (Ctrl+A, Delete), undid the deletion (Ctrl+Z). Reverting just this
fix and repeating the identical sequence reproduces the exact reported
error:
Debug: projectDataBase::addElement insert element info error :
QSqlError("1555", "Unable to fetch row",
"UNIQUE constraint failed: element_info.element_uuid")
With the fix, the same sequence produces nothing. Qt 6.10.2, ctest
13/13.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Use a bound VACUUM INTO path and remove the stale SQLite
handle declaration. Fix shell continuations in Windows CI and
Debian installation instructions.
A <graphics_table>'s <query> is stored in the .qet and executed when the
project loads. SQLite produces rows lazily, so the cost of that query is
not bounded by anything the project contains -- it is bounded by how long
the loop reading the rows is willing to run. A recursive CTE takes one line
to make that forever:
WITH RECURSIVE c(n) AS (SELECT 1 UNION ALL SELECT n+1 FROM c) SELECT n ...
Put that in the <query> of any project's summary table and opening the file
pins a core at 100% and grows ProjectDBModel::m_record until memory runs
out. Measured on examples/industrial.qet with the query swapped, built from
master:
clean --export-bom 3.6 s, 396 rows, exit 0
poisoned --export-bom killed at 90 s, still going, no output
No scripting, no MCP, no flag beyond an ordinary export. Opening the file in
the editor is the same code path.
QetScriptApi::query() has the identical loop, and the script engine's own
30 s interrupt does not reach it: that aborts JavaScript, and this is C++
inside a single call. Left alone it hung a --run for 45 s until the harness
killed it.
Both loops now stop at projectDataBase::MaxResultRows (100000) and say so.
That is a backstop, not a page size: the largest table in the shipped
examples is 396 rows, and a caller that reaches 100000 has been handed
something it should not run to completion. It is not silent either way --
the model logs the offending query text, and qet.query() sets queryError(),
so a truncated result is never mistaken for a complete one.
clean --export-bom 3.6 s, 396 rows, exit 0 (unchanged)
poisoned --export-bom 20.2 s, 396 rows, exit 0, warning names the query
qet.query(recursive CTE) 3.8 s, 100000 rows, queryError() set
Reverting each cap restores the hang, so both checks discriminate.
Related to #983, which fixes a different flaw reachable through the same
stored query. Neither depends on the other.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
projectDataBase::isReadOnlySelect() decides whether a query only reads
by looking at its first keyword and rejecting internal semicolons.
SQLite has allowed a CTE prefix in front of a data-modifying statement
since 3.8.3, so
WITH x AS (SELECT 1) DELETE FROM element
begins with WITH, contains no semicolon, passes the check, and deletes
every row. UPDATE and INSERT go through the same way.
This is not only reachable from the custom-query box. ProjectDBModel::
fromXml() reads a <graphics_table>'s saved <query> straight out of the
.qet and fillValue() executes it, so a project file can carry the
statement. Reproduced against a build of this branch's parent, with no
scripting and no CLI flag beyond the export itself: a project whose
stored table query was replaced with the DELETE above exported a bill
of materials of 0 rows instead of 14, exit code 0, nothing logged. A
silently empty or -- with UPDATE -- silently altered BOM is the kind of
output someone orders parts from.
Fixed by asking SQLite about the statement it actually compiled.
sqlite3_prepare_v2() compiles without running, sqlite3_stmt_readonly()
reports on the compiled statement rather than on how it was spelled,
and the prepare tail catches a second statement structurally. The same
project now exports its 14 rows again and logs a reason for the
refusal, while an ordinary WITH ... SELECT in a project file still runs
untouched -- the fix is not "ban CTEs".
isReadOnlySelect() stays in front of it rather than being replaced:
SQLite considers ATTACH, BEGIN and several PRAGMAs read-only too, since
none of them change the contents of the database, so dropping the
statement-type allowlist would have widened what is accepted while
fixing what is executed.
The check lives in its own translation unit depending on nothing but
QString and SQLite, so tests/qttest/tst_sqlreadonly.cpp can link it
alone and exercise the security property without standing up a
QETProject: 18 assertions covering the three CTE-prefixed writes named
in the review, bare writes, trailing statements, comment-only input
(which compiles to a null statement sqlite3_stmt_readonly() must not be
handed) and a null connection (refused, not waved through). Confirmed
the suite discriminates by deliberately disabling the new check and
watching exactly the nine write-refusal assertions go red while the
accept cases stayed green.
ctest 13/13, qet-coherence-check and qet-pdflink-check clean on the
example corpus.
Reported in PR #980's review thread by @elevatormind and confirmed
against this code by @scorpio810; fixed here on its own because the
flaw is in already-released code and needs none of that branch to
reach.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Refs discussion #886: a saved-report manager built on custom SQL and
nomenclature.json. Most of what was asked for already existed --
Projet > Exporter au format CSV already builds/saves/reuses named
SELECT queries via ElementQueryWidget and nomenclature.json, and
Projet > Ajouter une nomenclature already inserts one into a folio.
This fills the three real gaps.
- projectDataBase::isReadOnlySelect() rejects anything that isn't a
single SELECT/WITH statement. Checked in newQuery() itself, the one
choke point every query path already goes through -- including a
query loaded from a saved nomenclature/summary table's <query>
element on project open, not just the dialog's own custom-SQL box.
ElementQueryWidget shows the same check live as you type, and
BOMExportDialog surfaces it before running or exporting anything.
- BOMExportDialog gains a Preview button + table (QSqlQueryModel),
so a report can be checked on screen before committing to a CSV file.
- ElementQueryWidget gains Importer.../Exporter... buttons that
read/write nomenclature.json's saved reports as a JSON file, so a
report can be handed to a colleague or another install. Import asks
before overwriting a locally-saved report of the same name.
Verified: Qt 6.10.2, builds clean, ctest 8/8. Drove the real dialog
through Xvfb: typed "DROP TABLE element" into the custom-SQL box and
got the inline warning immediately, then confirmed Preview also
refuses it with a "Requête refusée" dialog rather than running it.
Preview against the real default query returned live column headers
and a row. Export opens a save dialog without crashing.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
ProjectDBModel::setQuery() calls projectDataBase::updateDB(), which drops
and repopulates every table in the database. The rebuild does not depend on
the query, so each table model that queries the database while a project is
being read triggers another complete repopulate of the same content.
Opening a 100 folio project ran updateDB() 26 times, 9.9 s of a 15.7 s load.
Two changes, because the first alone is not enough:
setUpdateBlocked() lets a bulk operation suppress the rebuild and do it
once when it is done. readProjectXml() already wrapped the load in
blockSignals(true) "to avoid hundreds of unnecessary emitted signal", but
that suppresses only the signal, not the work it announces; this extends
the same intent to the work. Both early returns in readProjectXml() sit
before the block, so no path leaves the database permanently blocked.
Further rebuilds are triggered after readProjectXml() returns, where the
load phase timers cannot see them -- with only the block in place
updateDB() still ran 5 times on examples/industrial.qet. So the database
now also tracks whether anything has changed since the last rebuild, and
skips repopulating when nothing has. dataBaseUpdated() is still emitted in
that case: callers and models rely on it to refresh, and what they read
back is the same either way. Every method of the class that writes rows
marks the flag; from outside, the database is reachable only through
newQuery(), and all five call sites read.
Repeating the rebuild was wasteful rather than wrong -- each
populate*Table() begins with a DELETE -- so this changes no output.
Verified byte identical --info on all 23 example projects, and identical
--export-bom, --export-wires, --export-cables, --export-nets and
--export-wiring on industrial.qet. Its load drops from 5.51 s to 5.31 s
(median of 6).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01L6MRq2Ach1ogvnGcbuqNLr
removeDiagram() only ever deleted the diagram's own row. No foreign key in
this schema is declared ON DELETE CASCADE (and SQLite foreign-key
enforcement is never turned on for this connection anyway), so removing a
diagram left every element, element_info, terminal and conductor row that
belonged to it behind in the database -- silently, since nothing reads them
until the next full updateDB() rebuild papers over it.
Traced why this had never crashed anything: Diagram::~Diagram() explicitly
walks and deletes its top-level items through removeItem() (which does call
dataBase()->removeElement() correctly), but deliberately skips conductors --
because a conductor's destructor touches both of its terminals
(terminal1->removeConductor(this)), and those terminals may belong to an
element already destroyed earlier in the same sweep. Conductors are instead
destroyed as a side effect of Terminal::~Terminal()'s qDeleteAll() on its own
conductor list, which is a plain C++ delete that never goes through
Diagram::removeItem() and therefore never calls dataBase()->removeConductor()
at all. So the object graph is torn down safely, but the database is never
told about the conductors or their terminals.
Fixed by adding the missing bulk deletes to projectDataBase::removeDiagram()
itself, run while the diagram (and its live scene) still exist -- verified
that QETProject::detachDiagram() emits diagramRemoved() (which this class's
constructor connects to this slot) synchronously, before the Diagram object
is scheduled for destruction via deleteLater(), so nothing here races the
C++ teardown described above. Order matters: element_info and terminal have
no diagram_uuid column of their own, so both are scoped through a subquery
on element and must run before element itself is deleted.
Verified against examples/industrial.qet (50 diagrams) by calling
projectDataBase::removeDiagram() directly and comparing table counts before
and after, with no intervening updateDB() call to mask a gap:
element=354->335 element_info=354->335 terminal=1087->1033 conductor=671->626 diagram=50->49
Every delta matches a direct SQL count for that diagram's own rows exactly
(19 elements, 54 terminals), and both "orphan rows still referencing the
removed diagram" checks read 0 afterward -- so the cascade is complete and,
just as importantly, scoped: nothing belonging to the other 49 diagrams
moved.
Separate finding, not fixed here: QETProject::removeDiagram(Diagram*) (the
synchronous, non-undoable variant, not the usual GUI
ProjectView::removeDiagram() path) segfaults if the enclosing QETProject is
destroyed before an event loop iteration lets its pending deleteLater() run
-- reproduces identically on unmodified master, so it predates and is
unrelated to this change. Worth its own report; a headless caller is the
only realistic way to hit it, which is how this surfaced.
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
The element and element_info tables had two independent insert paths --
addElement() for an element added to a live diagram, and
populateElementTable()/populateElementInfoTable() for a full rebuild --
which bound the same row differently. The incremental path wrote
kindInformations()["type"] into element.sub_type; the bulk path wrote
elementData().masterTypeToString(). So the table held different values
depending on whether the project had been reloaded since the element was
placed, and element_nomenclature_view exposes that column as
element_sub_type, which ElementQueryWidget filters on for the Coil,
Protection, Commutator and PLC nomenclature options.
That divergence is the same shape as the type-filter one fixed in the
previous commit, and it is the reason this stack kept finding bugs that
were invisible while editing and only appeared after a reload. Rather
than correct a second instance of it, both paths now go through
bindElementValues() and bindElementInfoValues(), following the
bindDiagramInfoValues() helper this class already had. Live and reloaded
now agree by construction instead of by coincidence.
The bulk path's values are the ones kept, because they are what every
already saved project contains: nothing a reload produces changes, and
the previous commit's 19-project BOM regression stays valid. It is the
live path that moves, onto the values a reload would have given it
anyway.
Measured, placing one element into a new project and then saving and
reopening it:
live element table: slave/ x1
reloaded element table: slave/ x1
and for the same element, what the two paths would have stored:
bulk (now shared): "" incremental (before this commit): "simple"
Re-ran the BOM regression over the same 19 projects after this change:
content identical to the pre-change baseline on all 19, and identical
line-for-line on 18, the exception being the three byte-identical
photovoltaique rows already described in the previous commit.
Note for anyone reading masterTypeToString(): the const no-argument
overload returns an empty string for anything that is not a Master, so
the "coil" fallback in the static overload is only reached for real
master elements. Non-master elements get an empty sub_type, not a
spurious "coil".
Slice 4 of discussion #503, on top of slice 3 (#629): the smallest
surface that makes wiring_list_view visible, plus the diagnostic the
view needs to be honest about what it is missing.
Projet > "Liste de câblage (base de données)" opens a read-only table of
wiring_list_view, headed by a line stating how many conductors are
listed and, when non-zero, how many were excluded and why.
Deliberately not another exporter. QET already ships a wiring-list CSV
export (Projet > Exporter le plan de câblage, and --export-cables) which
walks the project XML; measured on the same projects it produces a row
per conductor and resolves labels correctly when the project has them.
Adding a second, competing CSV would be worse, not better -- the
database path's value is what it unlocks (terminal plans, BOM joins),
not replacing that export.
projectDataBase::excludedConductorCount() counts, from the live scene,
the conductors deliberately absent from the conductor table because a
terminal has no uuid. Counted from the scene precisely because the
database is where those conductors are not. Verified: 671 on
examples/industrial.qet (which has 1794 terminals and no terminal uuids
at all, so its list is empty and now says so), 0 on a project whose
elements do carry terminal uuids.
KNOWN GAP, not fixed here and the reason this is opened for discussion
rather than merge: after a save/reload the component columns are blank
for slave elements. populateElementTable()/populateElementInfoTable()
only insert Simple|Terminal|Master|Thumbnail, so slave elements -- relay
contacts, i.e. a large share of real wire endpoints -- have no row in
element_info for the view to read a label from. Measured on a two-slave-
contact project after reload: element rows 0, element_info rows 0,
terminal rows 2, conductor rows 1; the wire is listed (slice 3's LEFT
JOIN keeps it) but both component names are empty, where the existing
CSV export shows K1 -> K2 for the same file.
Closing that gap means widening a filter shared with the nomenclature
and summary views, which would change what those existing, shipped
features contain. That is a maintainer decision, not one to take
unilaterally inside an additive slice.
Slice 3 of discussion #503, on top of slice 2 (#628). One row per
conductor, each endpoint resolved to its element label and terminal
name -- the `F1:4 -> M200:U1` shape from the original prototype.
The view deviates from the SQL sketched in the discussion in two ways,
both because the sketched version silently loses wires:
- **No join to the `element` table.** A terminal row already carries its
`element_uuid`, so joining `element` back just to read the same uuid
adds nothing. Worse, it filters: `populateElementTable()` only inserts
elements matching `Simple|Terminal|Master|Thumbnail`, so `Slave`
elements (relay contacts and the like -- extremely common at the end
of a wire) and report elements are simply absent from that table after
a project load, and an inner join through it drops their conductors.
- **`element_info` is LEFT joined** for the same reason. A wire whose
endpoint element has no info row still belongs in a wiring list; it
comes back with an empty label rather than vanishing. Losing a wire
from a wiring list is a worse failure than showing one with a blank
end.
Note this only bites after a save/reload. The incremental `addElement()`
path does not apply the type filter, so a slave element placed live is
present in `element`/`element_info` and an inner join looks fine -- it
is the bulk repopulate on project load that drops it. Testing only the
live-editing path would have missed this entirely.
Measured, comparing this view against an inner-join-through-element
variant built from the same tables in the same session:
| project | conductors | wiring_list_view | inner-join variant |
|---|---|---|---|
| Polonez MR'89 wiring diagram | 280 | 280 | 280 |
| two slave contacts, after save+reload | 1 | **1** | **0** |
Polonez happens to have no slave elements at conductor ends, so both
agree there and the problem is invisible. The second case is the
minimal reproduction: place two "Simple contact" elements
(`link_type="slave"`) so autoconnect wires them, save, reload -- the
sketched view returns zero rows for a project that plainly has a wire
in it.
Acceptance criterion held throughout: `wiring_list_view` row count
equals `conductor` row count, i.e. the view itself drops nothing.
Conductors already excluded upstream (legacy terminals without uuids,
see #628) stay excluded; that remains the only thing missing from the
list, and is what slice 4 should surface a count for.
Three fixes to the tables added by this slice.
A conductor's text was written once at insert and never again. Renaming a
wire left the database holding the old number, so the wiring list showed a
stale value until the next full repopulate -- elements have
elementInfoChanged() for exactly this and conductors had nothing.
Conductor::setProperties() has around a dozen call sites (auto-numbering,
the properties dialog, element moves, the delete command's re-links), so
rather than adding a call to each and missing the ones added later, listen
to the propertiesChange() signal it already emits. Qt::UniqueConnection
means a repeated insert or a full repopulate cannot double-subscribe, and
the connection is established on both insert paths because conductors read
from a file never pass through addConductor().
addConductor() and populateConductorTable() each carried their own copy of
the same seven bindValue() lines. They had not drifted yet, but that is the
same duplication the element paths had before bindElementValues(), where
they had drifted -- one binding kindInformations()["type"] and the other
masterTypeToString(). One bindConductorValues() for both.
Finally, index the conductor columns that get looked up per element rather
than per conductor. element_nomenclature_view counts the wires touching each
element with a correlated subquery, so without an index every element row
full-scans the conductor table and the cost grows as elements x conductors.
Measured on a standalone SQLite harness at 2000 elements x 5000 conductors:
2134 ms unindexed, 10 ms indexed. diagram_uuid is indexed too, since the
wiring list view joins on it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Slice 2 of discussion #503 (from-to wiring list built on projectDataBase),
building on the conductor uuid from slice 1 (#625). Pure plumbing: two
new additive tables plus their populate/add/remove hooks. No view, no UI,
no visible behavior change yet -- the wiring-list view is slice 3.
Follows the existing shape of the class throughout: same table/column
naming, same prepared-statement idiom in prepareQuery(), same
bind/exec/qDebug-lastError error handling, same DELETE-then-loop
populate pattern.
- `terminal (uuid, element_uuid, name)` and
`conductor (uuid, diagram_uuid, terminal1_uuid, terminal1_element_uuid,
terminal2_uuid, terminal2_element_uuid, text)` created alongside the
existing tables in createDataBase().
- populateConductorTable() added as a fifth populate* call in updateDB().
Terminal population is folded into it, since a terminal only matters
here in the context of a conductor referencing it.
- addConductor()/removeConductor() hooked into the already-existing
Conductor::Type branch of Diagram::addItem()/removeItem(), mirroring
the Element::Type branch directly above.
Two things the original schema sketch in the discussion got wrong, found
by testing rather than inspection:
1. Terminal::uuid() is NOT unique per placed terminal. It is the
terminal-position id baked into the catalog .elmt definition ("the
top terminal"), so every placed instance of the same catalog element
shares it. A terminal instance is only uniquely identified by
(uuid, element_uuid) together, so that pair is the terminal table's
primary key and the conductor table carries both halves for each
endpoint. With uuid alone as PK, the second placed instance of any
element silently lost its terminals to the INSERT OR IGNORE.
2. Conductors whose terminals predate terminal uuids are omitted rather
than given a fabricated identity, as agreed in the discussion. This
turns out to matter far more than expected in practice -- see below.
Testing (all live, in the running app):
- Incremental add: fresh project, two vertically aligned contacts placed
so autoconnect creates a conductor -> 2 terminals, 1 conductor.
- Incremental remove: deleting that conductor -> conductor count 1 -> 0.
- Undo: ctrl+Z after the delete -> back to 1, no duplicate-primary-key
error (the same Conductor object keeps its uuid).
- Bulk populate: examples/weneedpolonez-Polonez_MR89_wiring_diagram.qet
(366 conductors) -> 478 terminals, 280 conductors; the 86 conductors
touching legacy terminals correctly omitted.
- Join correctness: conductor -> terminal (composite key) -> element_info
resolves real from-to rows with real element labels.
- Legacy-only project: examples/industrial.qet has 1794 terminals and
*zero* terminal uuids, so all 671 of its conductors are omitted. Loads
and renders fine, no crash, no spurious rows -- but worth stating
plainly that a from-to wiring list for that project would be empty
today. This is a property of the element catalog definitions, not of
the project file, and is the strongest argument for surfacing an
"N conductors excluded" count to the user when the view lands.
- No SQL errors logged in any of the above.
Known limitation, consistent with existing behavior: removeDiagram()
does not cascade-delete the conductor rows of that diagram, exactly as
it already does not cascade to element/element_info. A full updateDB()
rebuild clears them, and the future wiring-list view INNER JOINs from
conductor, so orphan terminal rows never surface.
The export of internal database to file is now do by sqlite3 native
function instead of a dirty home made function.
For the moment, the export to file function is unavailable on macOSX (we
need some help to add sqlite3 lib on macOSX)
Magnus Hellströmer