The checks run once every folio is read. A .qetz written by this branch
has no keys in its tables, so a row twice is caught by the counts too.
A folio is copied as it was only from a database with the columns this
writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A folio file in a .qetz did not say which symbol each item is: the type
(embed://...) was moved to project.sqlite, so a folio read alone could
not be drawn or diffed. It stays in the folio file now; format 2 files,
which moved it, are still read.
Nothing checked that project.sqlite and the folio files agree: a row no
item has, an item's row on another folio, or information rows of no
symbol were read as far as they went. Reading now refuses such a file and
says what does not match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz held the engineering data twice: the document's tables, and
beside them QElectroTech's own project database (stage D2/D3), which
QElectroTech also builds from the document. Two copies of one fact can
disagree, and the copy cost a third of the file. Now project.sqlite holds
the document's tables only; QElectroTech builds its own project database
from them when the project opens, as it does for a .qet.
Measured (this build, lazy folios):
industrial (50 folios): 620 -> 374 KB, open 0.37 -> 0.65 s
f2099/nuri (136 folios): 3.27 -> 2.20 MB, open 2.2 -> 3.5 s
The same projects as .qet open in 3.3 s and 23 s.
Saving still copies the file of every folio not built, with its rows
(stage D4): the project keeps the part each folio was read from. nuri
opened and saved again: 6.3 s, no folio built, the same bytes.
A symbol's terminal list is left out of the folio file when every
folio, symbol and wire has a uuid of its own, now checked on the
document (QetContainer::ownUuids()) rather than by the stored database
being written.
Checked: 342 corpus projects open, .qetz -> .qet equals the plain
resave, a .qetz saved twice gives the same bytes; tst_lazyfolios,
tst_qetcontainer, tst_resaveunchanged (which now checks project.sqlite
holds only the document's tables).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1473 keys the .qetz database's rows by the items' uuids, stores
true/false as 1/0 and writes the format to PRAGMA user_version, as
format 3. This branch's own changes (computed text left out of the
folios, tables without keys, folios kept as they were) become format 4;
formats 2 and 3 are still read.
Folios kept as they were copy their rows by folio uuid, and the keys of
those rows are kept from the other items' keys. The document's tables
are named doc_<name> only beside the project database's own tables.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1470 keeps a .qetz's pictures out of the document; the index path still
reads them in, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers the points arummler raised in discussion #1440 about the
project.sqlite that .qetz files carry (format 2, PR #1462):
- Folio rows were keyed "f1", "f2"... by position, and every symbol and
wire carried a separate key beside its uuid. A row's key is now the
item's own uuid, which stays in the folio file; no key attribute is
added to the folio files of a project as QElectroTech saves it. An item
without a usable uuid (none, or one another item shares, F100) gets a
uuid made from its own content, the same on every save, never from
where it is.
- freezeLabel is stored as 0/1 (INTEGER) and an element information's
"show" in its own column; a value QElectroTech would not write stays in
the folio file verbatim. The tag is kept in the JSON only when it is
not the usual one.
- The database's PRAGMA user_version holds the format.
Format 3. Format 2 files, written by development builds since #1462,
still open: tst_qetcontainer reads one (fixtures/qetz_format2.qetz,
grafcet.qet saved by master aa503aec8) and saves it again.
Checked: tst_qetcontainer 38/38, and 343/343 readable projects of the
legacy corpus round-trip exactly. Every example saved as .qetz then .qet
equals its plain resave, and every format 2 file read by this build
equals what master writes from it. Each new test goes red with the
behaviour it checks broken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz re-saved after opening from its index saved wrong folio numbers
in its database: reading each folio's header wrote its row with a
number counted from the folios read so far ("1/1", "2/2" for "1/3",
"2/3"), and an index-opened project keeps the rows it was saved with,
so nothing wrote them again. The folio table's rows are no longer
written while the project is being read (projectDataBase::
diagramInfoChanged()). Folio list tables showed it.
A .qetz whose project.sqlite is not a database crashed QElectroTech: the
first statement could not be prepared and SQLite was handed a null
statement. A statement that cannot be prepared now binds nothing and
steps to no row, and the file is refused.
Gates: 342 corpus projects,
written by QElectroTech 0.3 to 0.200, all open; .qet -> .qetz -> .qet
equals the plain resave with every folio built; a .qetz re-saved twice
holds the same bytes entry by entry but the save's own name, path and
time (the first fault made the first re-save differ); hostile .qetz
(truncated, garbage database, view for a table, trigger, escaping and
repeated names, later format, missing folio, 2 GiB zip bomb) neither
crash nor hang and none opens; an unknown extra entry still opens.
tst_lazyfolios: folio numbers with totals from the index as when built
(fails without the fix); tst_qetcontainer: garbage database refused
(segfaults without it). ctest 96/96 both ways; save and export gates
same on 342.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Format 3 folio files lose what building a folio computes
(QetContainer::dropComputed()): a wire's description of its ends
(element1_label, element1_name, element1_linked, terminalname1 and the
same for end 2), the content of a symbol text showing the symbol's
information or a composition of it, and -- when the project database's
index is written, every symbol with a uuid of its own -- a symbol's list
of its terminals.
What may go was found by test, not by reading: on 342 projects saved by
this version, each kind taken out of the file, the project opened and
saved: the wire ends came back in all 244 that had them, the symbol texts
in all 201, the terminal lists in 305 of 306 -- not in the file whose
symbols share a uuid (#1408), where wires find their ends by them; such a
project gets no index, so it keeps them.
Stays, for now: labels made from a formula and wire numbers made from one,
in the database (opening from the index and numbering read them), and a
table's size (the user's as much as computed).
tst_qetcontainer compares a joined document with the original less what is
computed, and computedNotKept checks a file holding both kinds (one filled
in by the test) loses them -- it fails when nothing is dropped. Through
QElectroTech, 342 projects: a .qetz opened from its index and saved as .qet
= eager = master; every folio of industrial and f2099 drawn and linked as
eager (f2099 but its 3 render-noise folios). ctest 96/96 with and without
QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
project.sqlite now holds both what QElectroTech keeps in its project
database (the index D2 wrote as index.sqlite) and the document's
engineering data, the latter's tables named doc_<name> beside the
former's -- element, element_info, link and conductor exist in both
(QetContainerDb's SQL names them as before; Database::sql() puts the
prefix on). The manifest says format 3, min-reader 3: a format 2 reader
would find no tables of its names. Format 2 files, with or without an
index.sqlite entry, are read as before: the prefix is found from the
file (doc_meta).
A .qetz with an index opens from project.sqlite. Measured: the 136-folio
f2099 1.8 s, 5.2 MB (the two table sets still overlap: D3b).
Checked: tst_qetcontainer with the format 3 names, 34/34; a format 2 file
of the last commit opens from its index.sqlite and saves the same; corpus
342: 340 .qetz carry the index in project.sqlite and open from it, saved
back = eager = master on all 342. ctest 96/96 with and without
QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz keeps each folio picture as a PNG file. On opening, join() put
every picture back into the document as base64 text, and the folio then
decoded that text again, twice. On a project with 20 pictures of 7 MB
(140 MB of PNG, the shape of the project on PR #1462) that cost about
8 of the 12 seconds the open took:
QDomDocument::createTextNode() on the base64 3.9 s
DiagramImageItem::fromXml(), base64 decoded 3.9 s (2 x 1.9 s)
the PNG decoding itself 0.3 s
join() now takes an optional map. Given one, each picture stays a file:
the <image> keeps its qetz-file attribute and the bytes go in the map.
QETProject keeps the map for as long as the project is open, and
Diagram::fromXml() hands a picture's bytes to the new
DiagramImageItem::fromXml(element, picture). Kept, not cleared after
the load, so a folio built later finds its pictures too; the bytes are
shared with the PNG cache of the pictures built from them, so this costs
almost no memory. readDocument() without a map, as the automatic
numbering import uses it, still gives the whole document.
fromXml() of a .qet picture also decodes its base64 once instead of
twice: the first text node, which the PNG cache already used. e.text()
also collects <image_base>'s text, but loadFromData() stops at the end
of the first picture, so the picture shown is the same.
Measured, same file, 3 runs each, offscreen:
.qetz 11.8 s -> 3.8 s, peak memory 1034 MB -> 655 MB
.qet 15.7 s -> 13.6 s
Resaving either file with either binary gives the same bytes, a cropped
and colour-keyed picture included, and every folio exports to the same
PNG pixels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The project's terminal strips -- strip information, the terminals of
each physical terminal by level, bridges and what they bridge -- move to
four tables of project.sqlite, when rebuilding them from the rows gives
the same XML exactly (as QElectroTech writes them). Anything else, or
strips that carry text a save would not write, stays in project.xml.
Test: tst_qetcontainer::terminalStripsInDatabase (two strips with a
bridge and a two-level terminal on industrial.qet's own terminals: out
of project.xml, back exactly; stays in the XML when the rebuild is
broken). Outside ctest: a strip fixture saved as .qetz then .qet by
QElectroTech equals its straight resave, and the strips are rows any
SQLite tool reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Wga9B92BNurDEtxxtCDxf
Fourth stage of the zipped project file (#1440), format 2: what an
engineer would put in a list -- folio title-block fields and
properties, project properties, a symbol's uuid, type, prefix,
information and links, a wire's ends, number, function, colour,
section, cable and bus -- moves out of the XML into project.sqlite in
the .qetz, and lives only there. The folio files keep where things are
drawn. On opening it goes back into the document before the usual
reader runs, so nothing else in QElectroTech changes yet (the program
reading the database directly is the next step of the roadmap).
Each moved item keeps a key in the XML (qetz-key: its uuid where
unique); each moved block leaves a marker where it was, which keeps the
whitespace around it apart. A block of a shape this does not model
stays in the XML. A file is untrusted: the database is read with
trusted_schema off, query_only, and only from names that are plain
tables. Format 2 readers still open format 1; a format 1 reader
refuses format 2.
Tests (tst_qetcontainer): every example back exactly, whitespace
included; databaseHoldsTheData (industrial.qet: the folio files hold no
information, links, wire numbers or properties; the row counts match).
Outside ctest: 341 of 341 projects QET can open give the same .qet
through a format 2 .qetz.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Wga9B92BNurDEtxxtCDxf
Third stage of the zipped project file (#1440). A project is saved as a
.qetz when its file name ends in .qetz -- Save As offers it as a second
file type, and --resave, --set-titleblock, the script API's save() and
the live-mode snapshot follow the name too. Saving a .qet project still
writes a .qet; nothing changes unless the user picks .qetz.
Opening recognises a .qetz by its content (a zip), not its name, and
reads it as bytes: text mode would alter it on Windows. The project
document is joined back and read by the same code as a .qet. Crash
backups stay plain XML. Drag and drop, the open dialog and the
numbering import accept .qetz; the database export names its file from
the project's name without assuming ".qet".
Parts are now written without added indentation (one line each): the
symbol definitions are kept as read, whitespace included, and parts
that gained indentation made the next save differ (five forum projects).
Tests: tst_resaveunchanged::containerSavesTheSameProject (four examples
saved as .qetz, then as .qet, equal the straight .qet; pictures checked
to fail without the fix), tst_qetcontainer::noWhitespaceAdded. Outside
ctest: 341 of 341 projects QET can open (examples, bugtracker, GitHub,
forum) give the same .qet through a .qetz; 181.6 MB of .qet became
36.9 MB of .qetz.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Wga9B92BNurDEtxxtCDxf
Second stage of the zipped project file (#1440): QetContainer::split()
turns a project document into the entries of a .qetz -- mimetype first
and stored, a manifest with the format and the oldest format allowed to
read it, project.xml with each part replaced by <qetz-part file=.../>,
one file per folio, per symbol definition and per title block template,
and folio pictures as PNG files named by their SHA-256, stored once
each. join() puts it back and refuses a file of a later format, a
missing part or anything that is not a .qetz.
A picture leaves its folio only if its base64 comes back exactly, so
nothing here can change a project. Parts are written as QET writes a
.qet and read as it reads one. Still nothing opens or saves a .qetz.
Test: tst_qetcontainer, every example split, zipped, unzipped and
joined back to the same document (whitespace between elements aside);
pictures out of the folios once each; later format and missing part
refused. With QET_CONTAINER_CORPUS it also runs on a folder of projects:
341 of 341 readable projects collected from the bugtracker, GitHub and
the forum come back the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Wga9B92BNurDEtxxtCDxf
ispyisail