A folio is copied as it was only from a database whose folio_property and
element_info have the columns this writes too.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The JSON column of a property row held only its show flag, in every
project QElectroTech saves: the 25 examples and a 136-folio project used
it for nothing else. Folio and project properties now store it as a
number, as symbol information does, so a saved project's database holds
no JSON; JSON is left for attributes QElectroTech does not write (an
older or hand-edited file).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A symbol changed on a folio kept as it was needs the place its
information goes back to: its part is checked for the marker
(QetContainer::split()), where the position column said it before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The database held where each moved block sat among its parent's
children (props_at, info_at, links_at), though reading only asked
whether there was one: the marker the folio file keeps says that, and
where it goes back. The columns are gone; format 2 files leave the same
markers and still open.
A .qetz of a later format whose min-reader allows this version was
still refused by the database's own format check. It opens now,
read-only, with a message: saving it here could lose what this version
does not know. Save As writes a copy in this version's format.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The checks run once every folio is read. A .qetz written by this branch
has no keys in its tables, so a row twice is caught by the counts too.
A folio is copied as it was only from a database with the columns this
writes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A folio file in a .qetz did not say which symbol each item is: the type
(embed://...) was moved to project.sqlite, so a folio read alone could
not be drawn or diffed. It stays in the folio file now; format 2 files,
which moved it, are still read.
Nothing checked that project.sqlite and the folio files agree: a row no
item has, an item's row on another folio, or information rows of no
symbol were read as far as they went. Reading now refuses such a file and
says what does not match.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz held the engineering data twice: the document's tables, and
beside them QElectroTech's own project database (stage D2/D3), which
QElectroTech also builds from the document. Two copies of one fact can
disagree, and the copy cost a third of the file. Now project.sqlite holds
the document's tables only; QElectroTech builds its own project database
from them when the project opens, as it does for a .qet.
Measured (this build, lazy folios):
industrial (50 folios): 620 -> 374 KB, open 0.37 -> 0.65 s
f2099/nuri (136 folios): 3.27 -> 2.20 MB, open 2.2 -> 3.5 s
The same projects as .qet open in 3.3 s and 23 s.
Saving still copies the file of every folio not built, with its rows
(stage D4): the project keeps the part each folio was read from. nuri
opened and saved again: 6.3 s, no folio built, the same bytes.
A symbol's terminal list is left out of the folio file when every
folio, symbol and wire has a uuid of its own, now checked on the
document (QetContainer::ownUuids()) rather than by the stored database
being written.
Checked: 342 corpus projects open, .qetz -> .qet equals the plain
resave, a .qetz saved twice gives the same bytes; tst_lazyfolios,
tst_qetcontainer, tst_resaveunchanged (which now checks project.sqlite
holds only the document's tables).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1473 keys the .qetz database's rows by the items' uuids, stores
true/false as 1/0 and writes the format to PRAGMA user_version, as
format 3. This branch's own changes (computed text left out of the
folios, tables without keys, folios kept as they were) become format 4;
formats 2 and 3 are still read.
Folios kept as they were copy their rows by folio uuid, and the keys of
those rows are kept from the other items' keys. The document's tables
are named doc_<name> only beside the project database's own tables.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1470 keeps a .qetz's pictures out of the document; the index path still
reads them in, as before.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answers the points arummler raised in discussion #1440 about the
project.sqlite that .qetz files carry (format 2, PR #1462):
- Folio rows were keyed "f1", "f2"... by position, and every symbol and
wire carried a separate key beside its uuid. A row's key is now the
item's own uuid, which stays in the folio file; no key attribute is
added to the folio files of a project as QElectroTech saves it. An item
without a usable uuid (none, or one another item shares, F100) gets a
uuid made from its own content, the same on every save, never from
where it is.
- freezeLabel is stored as 0/1 (INTEGER) and an element information's
"show" in its own column; a value QElectroTech would not write stays in
the folio file verbatim. The tag is kept in the JSON only when it is
not the usual one.
- The database's PRAGMA user_version holds the format.
Format 3. Format 2 files, written by development builds since #1462,
still open: tst_qetcontainer reads one (fixtures/qetz_format2.qetz,
grafcet.qet saved by master aa503aec8) and saves it again.
Checked: tst_qetcontainer 38/38, and 343/343 readable projects of the
legacy corpus round-trip exactly. Every example saved as .qetz then .qet
equals its plain resave, and every format 2 file read by this build
equals what master writes from it. Each new test goes red with the
behaviour it checks broken.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Opened from its database, a .qetz's folios not built have no XML in
memory: reading each one's part only to look for wires referring to a
cable read every folio. The project database already knows which wires
carry a cable reference, and so which folios to build for the label pass.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The header declares projectDocument() returning a QDomDocument but did not
include it; it only built where the precompiled header pulled it in. CI
builds without one and failed.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The test interrupted QElectroTech at a random moment inside the dialog's
loop and called quitQET() there. Landing inside malloc(), the call waited
on the allocator's lock for ever: 1 run in 20 timed out at 120 s, on
master too. It now lets the program run on to the start of the main
thread's next QEventLoop::processEvents(), between two events, and calls
from there; the event loop watchdog wakes the loop every 50 ms.
A run could also stop before any dialog: SingleApplication keys a System V
shared memory block from a file in the temporary folder, 16 bits of the
key varying, and each run killed by gdb leaves its block; once a new
run's key fell on one of them (976 here), it refused to start. The run
now removes the blocks it leaves, attached to no process, and is tried
again (a new binary path, a new key, up to three times) when
SingleApplication stops it that way: the scenario never ran. The "dialog
entered" marker no longer prints when the program had exited.
60 runs under load, 976 old blocks present: 60 passed, one try repeated.
With refuseCloseWhileModal() disabled: 3 of 3 FAIL (signal 6).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A wire's properties are one row per attribute in the project database,
45 071 rows for f2099's 2 139 wires. The database a .qetz holds now keeps
only the attributes reading would not give anyway: each is left out in
turn when the properties read without it are the same (ConductorProperties
::fromXml(), its defaults); rows by wire and name, so the file does not
depend on memory order. Reading writes every attribute back into the
table, so queries and saves see what they saw before.
f2099: 9 387 rows, the file 3.60 MB -> 3.22 MB (5.14 MB before D3b);
industrial.qet 0.71 -> 0.61 MB. Gates with the master merge and the wire
edits by uuid: ctest 99/99 lazily, 98/99 with every folio built
(modal_quit_regression, the race fixed next); hostile .qetz all refused;
342 corpus projects save, export, round trip and re-save byte-stable.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Scripts get conductorPropertyByUuid() and setConductorPropertyByUuid(),
the wire named by its uuid (conductorUuids()) rather than by a symbol's
terminal. A change goes to every wire at the wire's potential, as
setConductorProperty() does. When those wires are all on folios not
built, none is numbered by a formula, and what changes in each is what a
.qetz database holds in its own columns (number, function, voltage,
colour and section of the wire, cable, bus), it goes through the project
database: the store and the conductor rows, undoable by uuid. The
potential is followed in the database (projectDataBase::potentialWires():
through wire ends, folio reports and terminal blocks, as
Conductor::relatedPotentialConductors() follows it; a terminal block that
isolates its terminals answers nothing). Building a wire takes its change
(Conductor::fromXml()); a .qetz save writes it in the database, the folio
file copied; a folio written from its XML gets it there; a .qet builds.
Anything else builds the folio, as before.
tst_lazyfolios: every wire's potential from the database = as followed,
1 342 of 1 342 (industrial, eager and from a .qetz); a cable set by uuid
on a 14-wire potential builds no folio, reaches every wire, saves as when
built, undoes, shows once built (fails when building ignores the change,
and when the .qetz writer does). The test's run limit is raised (500 s ->
1 500 s, ctest 1 800 s): under load it ran past 500 s. ctest 96/96 both
ways.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every tool now takes a .qetz as it takes a .qet. The tools that read a
project's XML read the .qet the configured QElectroTech saves from the
.qetz, made once per version of the file (path, size, modification
time); without a QElectroTech, they say to set QET_BINARY. The tools that
launch QElectroTech hand it the .qetz, and write one when the output path
ends in .qetz. qet_scan counts .qetz files too. The server's description
and README say so.
Reading through QElectroTech instead of the file's own layout keeps the
server right while the zipped format still changes; the first read of a
big .qetz takes as long as QElectroTech needs to save it.
test_qet_mcp.py: the read tools answer the same for a .qetz and its .qet,
qet_diff finds nothing between them, qet_scan counts both, a .qetz is
converted once, qet_query reads it, qet_edit writes one (four fail when
_root() parses the file itself); without QElectroTech the answer names
QET_BINARY. Whole suite: 430 tests pass with QET_ENABLE_SCRIPTING=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A query of drawing_item_view, which is made of the shape, text and
picture tables, built no folio, and with folios built only when needed
(the default since D6) it found nothing: the drawn items are written to
the database by built folios. A query naming the view builds them, as
one naming the tables does.
Found by the qet MCP server's suite (four drawn-item tests failed lazily,
passed with every folio built). tst_lazyfolios: the view's rows from a
.qetz as when built (5 = 5).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The database a .qetz holds is read whole: its tables are now written
without their primary keys and indexes, in the order of their rows, and
the file compacted; the reader indexes what it looks folios up by once
it has read the file. Keys and indexes on text uuids were half of the
database once zipped. A file written before reads as before.
f2099 (136 folios) 5.14 MB -> 3.60 MB, industrial.qet 1.03 -> 0.71 MB;
opening and showing a folio take as long as before (2.5 s, 215 ms). Gates
on 342 corpus projects: saves, exports, round trip, re-save byte-stable,
all open; hostile .qetz all refused; ctest 96/96 both ways.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The strings the zipped project format added (Save a copy for an older
QElectroTech, the notice when a .qet is saved as .qetz beside it, the
.qetz file filters) and the undo text of a symbol's information changed
through the project database. Only French, by hand, in the wording the
file already uses ("Projets QElectroTech (*.qet)", "Modifier les
informations de l'élément", "Fichier"); the other languages are left for
their translators. lrelease: 0 unfinished; lupdate over sources keeps
every entry finished.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz re-saved after opening from its index saved wrong folio numbers
in its database: reading each folio's header wrote its row with a
number counted from the folios read so far ("1/1", "2/2" for "1/3",
"2/3"), and an index-opened project keeps the rows it was saved with,
so nothing wrote them again. The folio table's rows are no longer
written while the project is being read (projectDataBase::
diagramInfoChanged()). Folio list tables showed it.
A .qetz whose project.sqlite is not a database crashed QElectroTech: the
first statement could not be prepared and SQLite was handed a null
statement. A statement that cannot be prepared now binds nothing and
steps to no row, and the file is refused.
Gates: 342 corpus projects,
written by QElectroTech 0.3 to 0.200, all open; .qet -> .qetz -> .qet
equals the plain resave with every folio built; a .qetz re-saved twice
holds the same bytes entry by entry but the save's own name, path and
time (the first fault made the first re-save differ); hostile .qetz
(truncated, garbage database, view for a table, trigger, escaping and
repeated names, later format, missing folio, 2 GiB zip bomb) neither
crash nor hang and none opens; an unknown extra entry still opens.
tst_lazyfolios: folio numbers with totals from the index as when built
(fails without the fix); tst_qetcontainer: garbage database refused
(segfaults without it). ctest 96/96 both ways; save and export gates
same on 342.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every project now opens without building its folios; a folio is built
when it is shown or something needs its items (QETProject::lazyFolios()).
QET_LAZY_FOLIOS is gone; QET_EAGER_FOLIOS=1 builds every folio on
opening, as before, for comparisons and troubleshooting. Comments that
named the old variable now say "lazy folios"; the tests switch modes
with the new one.
The first save of a .qet as .qetz still builds every folio: a .qet does
not say which program wrote it, so its folios are written again rather
than copied. A .qetz opens and saves without building.
ctest 96/96 by default and with QET_EAGER_FOLIOS=1; 342 corpus projects
save the same in all four arms and export the same. In the GUI
(qet-gui-dialog): a 136-folio .qetz opens with no folio built, shows
folios 1 to 4, saves building none, the save = the original but the time
spent in it; a .qet opens lazily, saves as a .qetz beside it after the
notice, the .qet unchanged.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A folio not built whose own data changed was built to save, with the
folios its links reach. When what changed is only title block fields
nothing on a folio is computed from -- its title, author, date,
revision index, version, file name, template; a formula reads the
folio's label, plant, location, extra fields and grid -- its items as
read are what building it would write: it is saved from them with its
own data as it is now (Diagram::deferredXmlWithHeader()), .qet and
.qetz alike, with the information of its symbols changed through the
project database (or built, when one of those needs it). Any other
change still builds it.
136-folio .qetz, revision index and author set on every folio by a
script: 0.8 s, saved in 6.8 s, no folio built; with every folio built,
the same file. tst_lazyfolios: a new revision on every folio of a .qet
and of a .qetz saved as when built, building only the folios holding a
table (fails, 47 built, when every change counts); a folio whose label
changed is built (fails when none counts). ctest 96/96 eager and lazy;
342 corpus projects save the same in all four arms and export the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A script changing the information of a symbol on a folio not built, when
nothing else shows that information (no symbol built holds it, no
partner is linked to it, no label formula), changes it through the
project database: the store and the symbol's element info row, undoable
by uuid (ChangeStoredElementInformationCommand). Reading it back reads
the store. Building the symbol later takes the change
(projectDataBase::takeUnbuiltEdit(), in Element::fromXml()), the label
and texts following as for any change. Saving a .qetz writes the
changed information in the database in place of the rows the folio was
read with, its file copied, the folio still not built; a symbol whose
file holds no information block, or a .qet, builds the folio to save.
136-folio .qetz, "comment" set on the 3 513 symbols the database can
change: 0.9 s, no folio built, saved in 6.1 s, none built (27 s and 116
folios when saving built them). The same edits on the 4 039 symbols with
no partner and no formula, lazily and with every folio built: the two
saved projects are the same file. tst_lazyfolios: a script changes a
symbol not built, building none; saved as when built (fails when
building ignores the change, and when the .qetz writer does); undone;
the folio built afterwards shows it. ctest 96/96 eager and lazy; 342
corpus projects save the same in all four arms and export the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scripting API (scripts and the assistant's live mode) reached a folio
through diagrams(), which builds every folio, for every call. A folio's
own data -- count, title, uuid, title block, border, wire defaults --
now comes from folios(), built or not; a call reading or changing a
folio's items builds that folio, as showing it does (built()); selecting
a symbol by uuid builds the folio the project database places it on.
Search and replace and the continuity check still build every folio.
DiagramContent worked out one wire of each potential when made, which
follows potentials through folio reports and built the folios at the
other end, for every caller; it is now worked out when first asked
(Search and replace asks; a merge asks both sides, as before).
tst_lazyfolios: on a .qetz, a script reads every folio's title, uuid and
border building none (fails with diagrams()), and the symbols and wires
of the folio with the most links to other folios as when built, building
that folio only (27 before potentials were lazy). ctest 96/96 but
modal_quit_regression in both modes: a race in that test (gdb interrupts
the main thread inside malloc while the folio view repaints during the
dialog, master too), 2 of 23 runs here, 0 of 10 on master, passing alone.
342 corpus projects save the same in all four arms, and export the same
wiring list and wire numbers lazily.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The wiring list export read the project as QETProject::toXml() writes
it, which builds every folio. It now reads a document made without
building: a built folio written as saving writes it
(QETProject::folioXml(), which keeps its symbols' links to partners not
built), a folio not built copied from its XML, and what QElectroTech
computes for a wire's ends -- its symbols' labels, their last partner's
label, the terminals' names -- filled in from the symbols built and from
the project database. A symbol still waiting for a partner not built
takes its partners from the database, saved order.
136-folio .qetz, wiring list from the command line: 30.1 s with every
folio built, 4.5 s lazily, same file. tst_lazyfolios: the wiring list of
a .qet and of a .qetz as when built, building no folio. New corpus gate
(it fails on
all 6 projects tried with terminal names broken): --export-cables and
--export-wires lazily from the .qet and the .qetz = eager on 342
projects; one (forum f3386) differed until waiting symbols took their
partners from the database. ctest 96/96 eager and lazy (one earlier
run had modal_quit_regression and tst_resaveunchanged fail once each,
both passing on rerun in both modes); 342 corpus projects save the same
in all four arms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Terminal numbering builds only the folios holding a terminal. The wire
numbers export reads the wires of the folios not built from the project
database, and walks a built folio's own wires instead of a
DiagramContent, which followed their potentials through folio reports
and built the folios at the other end. The wiring list export names a
folio from its title block, which a folio not built has; it still
builds every folio, writing the whole project out (toXml()).
tst_lazyfolios: wire numbers exported as when built, from a .qet and a
.qetz, building no folio (fails with DiagramContent: 27 built);
terminal numbering builds 7 folios, those with a terminal (fails with
diagrams(): 47); the wiring list as when built. ctest 96/96 eager and
lazy; 342 corpus projects save the same in all four arms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adding, moving or removing a folio rewrote every folio's position in
the project database through diagrams(), which builds every folio: 3 s
on a 50-folio project opened lazily. A folio not built has its position
and title block already, so folios() is enough. The list of tables a
table can continue builds only the folios holding a table (from the
index of a .qetz), and the wiring list's count of wires left out looks
at built folios only: reading a wire drops it unless both its ends are
found.
tst_lazyfolios: a folio added, moved and removed builds no other, with
the folios' rows as when every folio is built (fails with diagrams():
47 built); the tables listed as when built, building the 12 folios that
hold one (fails when every folio is built). ctest 96/96 eager and lazy;
342 corpus projects save the same in all four arms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The project database's element table now holds each symbol's definition
(the path its file names it by) and its sequential numbers, filled from
the document and from built folios alike. Numbering, the searches for
symbols of a kind (a link tab's candidates) and the list of unused
symbols read them for the folios not built, instead of reading those
folios' files. Saving looks up a folio's tables in the index it was
opened from.
A .qetz asked its numbering, searches and unused symbols reads only the
files of the folios it builds (tst_lazyfolios, which fails when the
unused symbols are read from the folio files: 50 read, 22 built).
ctest 96/96 eager and lazy; 342 corpus projects save the same in all
four arms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Showing a folio built it and every folio holding a partner of its
symbols. Now a partner is built alone, as a symbol, into its folio's
scene (Diagram::buildSymbols()), and reused when that folio is built.
- A linked folio report built alone had no wires when it was linked,
so its texts showing a wire's properties stayed empty: after any
build, linked reports on the built folios and their partners look at
their wires again, as opening does. Unlinked reports keep their texts,
as opening leaves them.
- A report whose partner was built alone builds the partner's folio
when its potential is followed.
- tableChainFolios() parsed every unbuilt folio's file. The index now
lists the drawn tables and the table each continues (folio_table),
and a .qetz opened from it uses that list. An index without it falls
back to the folio files.
- A .qetz's project.sqlite is opened once per project
(QetContainerDb::FolioReader), not once per folio built.
nuri (136 folios), opened fresh, one folio shown: median 1104 -> 184 ms,
each show builds one folio; industrial 234 -> 87 ms.
tst_lazyfolios: a folio whose table continues another is shown first
from a .qetz (fails when the index's chains are ignored). ctest 96/96
eager and lazy; 342-project corpus: master, lazy, .qetz lazy and .qetz
kept all the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A built symbol whose partner is on a folio not built yet (QET_LAZY_FOLIOS)
now saves that link too, in the order it was saved with
(Element::toXml(); QETProject::waitingLinkTargets() keeps it to partners
the project has, so a link to a symbol gone is dropped as it always was).
With that, and with no partner's text in a .qetz's folio files (D3c),
saving a .qetz no longer builds the folios of the partners built folios
wait for; table chains still are (a table saves the table before it), and
a .qet saved as read (QET_LAZY_SAVE_KEPT) still builds them, its folio
files holding partners' texts.
industrial.qetz: an untouched save builds no folio (47 of 47 stay not
built; 19 before), after everyday edits on its busiest folio 29 stay not
built (7 before).
tst_lazyfolios compares the links an edited .qetz saved without building
holds, symbol by symbol in their order, with those the same edits saved
with every folio built -- it fails when waiting links are not saved (the
drawing alone cannot tell: links are kept on both sides, and a symbol
reread gets its partner's back, in another order). Corpus, 342 projects:
a .qetz saved without building, read and saved as .qet = eager = master.
ctest 96/96 with and without QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With QET_LAZY_FOLIOS=1, a project opened from a .qetz's index saves a
.qetz without building the folios it never built, when building them
would write the same: their parts are copied as they were and their rows
of the document's engineering data from the project.sqlite they were read
from (QetContainerDb::extract() with kept_from, QetContainer::split() with
the old entries, pictures included). Whether a folio may be kept follows
the rules of stage 5.5 (QETProject::toXml() with kept_parts): nothing
project-wide changed since opening; a folio whose own data changed is
built with its partners; after an edit, folios holding a table filled
from the database are built; the partners still waiting of every built
folio are built. Trusted only for a project opened from a format 3 index
written by this version whose folios are named by their uuid.
Format 3 names a folio by its uuid in the database (doc_meta folio_ids =
uuid) when every folio has one of its own, so that a kept folio keeps its
rows whatever its place; copies keep the rows' order (ORDER BY rowid).
A folio not built keeps the name of its part until it is built
(Diagram::deferredPart()).
Measured, the 136-folio f2099 opened from its .qetz and saved as a .qetz:
4.75 s with no folio built, 33 s with every folio built; its 136 folio
parts copied byte for byte; a kept save of a kept save is the same bytes.
industrial.qetz untouched: 28 of its 47 folios not built are built to save
(link chains, as in 5.5); after everyday edits on its busiest folio, 7
stay not built.
tst_lazyfolios: an untouched .qetz saved without building, read back,
holds the project; edits saved without building hold, read back, what the
same edits saved with every folio built hold (read back too: a wire
numbered from its potential is saved with no sequential numbers whichever
the way -- reading it back renumbers it, a separate issue); fails when a
kept folio's symbol information is not copied. Corpus, 342 projects: a
.qetz saved without building, read and saved as .qet = eager = master on
all 342. ctest
96/96 with and without QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Format 3 folio files lose what building a folio computes
(QetContainer::dropComputed()): a wire's description of its ends
(element1_label, element1_name, element1_linked, terminalname1 and the
same for end 2), the content of a symbol text showing the symbol's
information or a composition of it, and -- when the project database's
index is written, every symbol with a uuid of its own -- a symbol's list
of its terminals.
What may go was found by test, not by reading: on 342 projects saved by
this version, each kind taken out of the file, the project opened and
saved: the wire ends came back in all 244 that had them, the symbol texts
in all 201, the terminal lists in 305 of 306 -- not in the file whose
symbols share a uuid (#1408), where wires find their ends by them; such a
project gets no index, so it keeps them.
Stays, for now: labels made from a formula and wire numbers made from one,
in the database (opening from the index and numbering read them), and a
table's size (the user's as much as computed).
tst_qetcontainer compares a joined document with the original less what is
computed, and computedNotKept checks a file holding both kinds (one filled
in by the test) loses them -- it fails when nothing is dropped. Through
QElectroTech, 342 projects: a .qetz opened from its index and saved as .qet
= eager = master; every folio of industrial and f2099 drawn and linked as
eager (f2099 but its 3 render-noise folios). ctest 96/96 with and without
QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
project.sqlite now holds both what QElectroTech keeps in its project
database (the index D2 wrote as index.sqlite) and the document's
engineering data, the latter's tables named doc_<name> beside the
former's -- element, element_info, link and conductor exist in both
(QetContainerDb's SQL names them as before; Database::sql() puts the
prefix on). The manifest says format 3, min-reader 3: a format 2 reader
would find no tables of its names. Format 2 files, with or without an
index.sqlite entry, are read as before: the prefix is found from the
file (doc_meta).
A .qetz with an index opens from project.sqlite. Measured: the 136-folio
f2099 1.8 s, 5.2 MB (the two table sets still overlap: D3b).
Checked: tst_qetcontainer with the format 3 names, 34/34; a format 2 file
of the last commit opens from its index.sqlite and saves the same; corpus
342: 340 .qetz carry the index in project.sqlite and open from it, saved
back = eager = master on all 342. ctest 96/96 with and without
QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz now carries index.sqlite: the project database as QElectroTech
keeps it -- every table but the drawing items, which exist for built
folios only -- with each folio's own data (Diagram::headerXml()) and a
stamp of the schema it was written with (projectDataBase::indexFile(),
QETProject::writeContainerFile(): Save, --resave, --set-titleblock, a
script's and live mode's save). Written when every folio, symbol and wire
has a uuid of its own; a reader that does not know the entry skips it.
With QET_LAZY_FOLIOS=1, a .qetz with an index opens from it: the project
document is joined without its folios (QetContainer::join() with
folio_files), the database and the stores of symbol information and wire
properties are filled from the index (fillFromIndex()), each folio is made
from its own data and keeps the name of its part, read when the folio is
built (Diagram::deferredXml(), QETProject::readFolioPart(),
QetContainer::readFolio() -- pictures and engineering data back from
project.sqlite, QetContainerDb::restoreFolio()). A terminal strip finds its
terminals' folios in the database. Any other .qetz, or one whose index
cannot be used, is read whole as before.
The index is untrusted: refused unless written with this database's
schema, every table read a plain table (not a view), nothing in its schema
run (trusted_schema off); refused, the project is read whole. Checked: an
index of garbage, a wrong stamp, a table made a view -- each refused and
the project saved as a plain save.
The 136-folio f2099 as .qetz opens in 1.8 s (3.65 s read whole, 29 s
building every folio); what is left is mostly its 7 MB of symbol
definitions (0.8 s). The file grows from 2.6 to 5.2 MB, the index
overlapping project.sqlite until D3.
tst_lazyfolios: a .qetz opened from its index builds only the terminal
strip's folios, numbering read as when built, every folio shown in turn
draws and links as when built, the save as when built; fails when a folio
part's engineering data is not put back. Corpus, 342 projects: 340 .qetz
carry an index and open from it (2 have symbols sharing a uuid); saved
back as .qet = eager = master on all 342. Probe on industrial and f2099
.qetz: every folio as eager but the 3 render-noise folios of f2099. In a
real window the .qetz opens and pages building only shown folios and
partners. ctest 96/96 with and without QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The zipped project (.qetz) is the format QElectroTech saves.
- Save As and the first Save of a new project offer .qetz first
("untitled.qetz"); .qet can still be chosen.
- A project read from a .qet is saved as a .qetz beside it, the .qet left
as it is (QETProject::write(), so Save, autosave, a script's save and the
assistant's live save all do it). The project then works on the .qetz.
A path chosen afterwards (Save As, a script naming one) is saved to as it
is. The editor says so before the first such save, and asks before
replacing a .qetz already there.
- File > Save a copy for an older QElectroTech writes a .qet copy, for a
version that cannot open a .qetz; the project keeps its own file.
- The operating systems know the file: a freedesktop MIME type
application/x-qelectrotech-project+zip (*.qetz, and by its mimetype
entry when the name says nothing), the desktop file, macOS's document
types and the Windows installer's association.
Crash backups stay XML, written beside the project as before.
tst_resaveunchanged::qetSavedAsQetzBeside: a .qet saved where it is leaves
the .qet byte for byte and writes a .qetz beside it holding what a plain
save of the .qet holds; it fails when the switch is off. In a real window:
Ctrl+S on perceuse.qet shows the notice, writes perceuse.qetz and leaves
perceuse.qet's md5 unchanged; the copy for an older version (reached by
command search) equals a plain save but for time_spent; a new project's
Save offers untitled.qetz. xdg-mime finds the type by name and by content.
ctest 96/96 with and without QET_LAZY_FOLIOS=1; srclint R-C clean.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Comments added with the lazy folios cited planning notes kept outside the
QElectroTech repository, which a reader of the source cannot open. The
sentences now stand on their own.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With QET_LAZY_FOLIOS=1, a project whose database cannot be filled from
the file (folios or symbols with no saved uuid, two symbols sharing one,
a folio number using %autonum, wires naming their ends the older way)
had its folios left unbuilt, then built all at once while the database
was filled from them. That is slower than building them as they are
read: a 17 MB project reported on the pull request took 24.8 s to open
instead of 19.2 s, and most files saved by an older version have no
folio uuids.
The file is now checked for those reasons before the folios are read;
when one applies the project opens as without QET_LAZY_FOLIOS. The rarer
reasons, which need the definitions or every folio's links, still fall
back as before.
tst_lazyfolios opens a copy of industrial.qet with one symbol's uuid
given to another, and checks no folio is built after reading.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz keeps each folio picture as a PNG file. On opening, join() put
every picture back into the document as base64 text, and the folio then
decoded that text again, twice. On a project with 20 pictures of 7 MB
(140 MB of PNG, the shape of the project on PR #1462) that cost about
8 of the 12 seconds the open took:
QDomDocument::createTextNode() on the base64 3.9 s
DiagramImageItem::fromXml(), base64 decoded 3.9 s (2 x 1.9 s)
the PNG decoding itself 0.3 s
join() now takes an optional map. Given one, each picture stays a file:
the <image> keeps its qetz-file attribute and the bytes go in the map.
QETProject keeps the map for as long as the project is open, and
Diagram::fromXml() hands a picture's bytes to the new
DiagramImageItem::fromXml(element, picture). Kept, not cleared after
the load, so a folio built later finds its pictures too; the bytes are
shared with the PNG cache of the pictures built from them, so this costs
almost no memory. readDocument() without a map, as the automatic
numbering import uses it, still gives the whole document.
fromXml() of a .qet picture also decodes its base64 once instead of
twice: the first text node, which the PNG cache already used. e.text()
also collects <image_base>'s text, but loadFromData() stops at the end
of the first picture, so the picture shown is the same.
Measured, same file, 3 runs each, offscreen:
.qetz 11.8 s -> 3.8 s, peak memory 1034 MB -> 655 MB
.qet 15.7 s -> 13.6 s
Resaving either file with either binary gives the same bytes, a cropped
and colour-keyed picture included, and every folio exports to the same
PNG pixels.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
miniz is built with MINIZ_NO_TIME so that saving the same project twice
gives the same bytes. Without a clock it writes 0 as each entry's DOS
date: day 0 of month 0, 1980, which is not a date. Unzip tools show it
as 30 Nov 1979, others fail the conversion and show 13 Dec 1901
(reported on PR #1462).
Write 1 Jan 1980 00:00, the first valid DOS date, into the local and
central header of every entry once the archive is built. The bytes stay
reproducible, and miniz stays unmodified. Using the save time instead
was rejected: two saves would no longer give the same file, and miniz
converts through the local time zone, so even a fixed time_t would
differ between machines.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ispyisail