The database a .qetz holds is read whole: its tables are now written
without their primary keys and indexes, in the order of their rows, and
the file compacted; the reader indexes what it looks folios up by once
it has read the file. Keys and indexes on text uuids were half of the
database once zipped. A file written before reads as before.
f2099 (136 folios) 5.14 MB -> 3.60 MB, industrial.qet 1.03 -> 0.71 MB;
opening and showing a folio take as long as before (2.5 s, 215 ms). Gates
on 342 corpus projects: saves, exports, round trip, re-save byte-stable,
all open; hostile .qetz all refused; ctest 96/96 both ways.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz re-saved after opening from its index saved wrong folio numbers
in its database: reading each folio's header wrote its row with a
number counted from the folios read so far ("1/1", "2/2" for "1/3",
"2/3"), and an index-opened project keeps the rows it was saved with,
so nothing wrote them again. The folio table's rows are no longer
written while the project is being read (projectDataBase::
diagramInfoChanged()). Folio list tables showed it.
A .qetz whose project.sqlite is not a database crashed QElectroTech: the
first statement could not be prepared and SQLite was handed a null
statement. A statement that cannot be prepared now binds nothing and
steps to no row, and the file is refused.
Gates: 342 corpus projects,
written by QElectroTech 0.3 to 0.200, all open; .qet -> .qetz -> .qet
equals the plain resave with every folio built; a .qetz re-saved twice
holds the same bytes entry by entry but the save's own name, path and
time (the first fault made the first re-save differ); hostile .qetz
(truncated, garbage database, view for a table, trigger, escaping and
repeated names, later format, missing folio, 2 GiB zip bomb) neither
crash nor hang and none opens; an unknown extra entry still opens.
tst_lazyfolios: folio numbers with totals from the index as when built
(fails without the fix); tst_qetcontainer: garbage database refused
(segfaults without it). ctest 96/96 both ways; save and export gates
same on 342.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A script changing the information of a symbol on a folio not built, when
nothing else shows that information (no symbol built holds it, no
partner is linked to it, no label formula), changes it through the
project database: the store and the symbol's element info row, undoable
by uuid (ChangeStoredElementInformationCommand). Reading it back reads
the store. Building the symbol later takes the change
(projectDataBase::takeUnbuiltEdit(), in Element::fromXml()), the label
and texts following as for any change. Saving a .qetz writes the
changed information in the database in place of the rows the folio was
read with, its file copied, the folio still not built; a symbol whose
file holds no information block, or a .qet, builds the folio to save.
136-folio .qetz, "comment" set on the 3 513 symbols the database can
change: 0.9 s, no folio built, saved in 6.1 s, none built (27 s and 116
folios when saving built them). The same edits on the 4 039 symbols with
no partner and no formula, lazily and with every folio built: the two
saved projects are the same file. tst_lazyfolios: a script changes a
symbol not built, building none; saved as when built (fails when
building ignores the change, and when the .qetz writer does); undone;
the folio built afterwards shows it. ctest 96/96 eager and lazy; 342
corpus projects save the same in all four arms and export the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Showing a folio built it and every folio holding a partner of its
symbols. Now a partner is built alone, as a symbol, into its folio's
scene (Diagram::buildSymbols()), and reused when that folio is built.
- A linked folio report built alone had no wires when it was linked,
so its texts showing a wire's properties stayed empty: after any
build, linked reports on the built folios and their partners look at
their wires again, as opening does. Unlinked reports keep their texts,
as opening leaves them.
- A report whose partner was built alone builds the partner's folio
when its potential is followed.
- tableChainFolios() parsed every unbuilt folio's file. The index now
lists the drawn tables and the table each continues (folio_table),
and a .qetz opened from it uses that list. An index without it falls
back to the folio files.
- A .qetz's project.sqlite is opened once per project
(QetContainerDb::FolioReader), not once per folio built.
nuri (136 folios), opened fresh, one folio shown: median 1104 -> 184 ms,
each show builds one folio; industrial 234 -> 87 ms.
tst_lazyfolios: a folio whose table continues another is shown first
from a .qetz (fails when the index's chains are ignored). ctest 96/96
eager and lazy; 342-project corpus: master, lazy, .qetz lazy and .qetz
kept all the same.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
With QET_LAZY_FOLIOS=1, a project opened from a .qetz's index saves a
.qetz without building the folios it never built, when building them
would write the same: their parts are copied as they were and their rows
of the document's engineering data from the project.sqlite they were read
from (QetContainerDb::extract() with kept_from, QetContainer::split() with
the old entries, pictures included). Whether a folio may be kept follows
the rules of stage 5.5 (QETProject::toXml() with kept_parts): nothing
project-wide changed since opening; a folio whose own data changed is
built with its partners; after an edit, folios holding a table filled
from the database are built; the partners still waiting of every built
folio are built. Trusted only for a project opened from a format 3 index
written by this version whose folios are named by their uuid.
Format 3 names a folio by its uuid in the database (doc_meta folio_ids =
uuid) when every folio has one of its own, so that a kept folio keeps its
rows whatever its place; copies keep the rows' order (ORDER BY rowid).
A folio not built keeps the name of its part until it is built
(Diagram::deferredPart()).
Measured, the 136-folio f2099 opened from its .qetz and saved as a .qetz:
4.75 s with no folio built, 33 s with every folio built; its 136 folio
parts copied byte for byte; a kept save of a kept save is the same bytes.
industrial.qetz untouched: 28 of its 47 folios not built are built to save
(link chains, as in 5.5); after everyday edits on its busiest folio, 7
stay not built.
tst_lazyfolios: an untouched .qetz saved without building, read back,
holds the project; edits saved without building hold, read back, what the
same edits saved with every folio built hold (read back too: a wire
numbered from its potential is saved with no sequential numbers whichever
the way -- reading it back renumbers it, a separate issue); fails when a
kept folio's symbol information is not copied. Corpus, 342 projects: a
.qetz saved without building, read and saved as .qet = eager = master on
all 342. ctest
96/96 with and without QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
project.sqlite now holds both what QElectroTech keeps in its project
database (the index D2 wrote as index.sqlite) and the document's
engineering data, the latter's tables named doc_<name> beside the
former's -- element, element_info, link and conductor exist in both
(QetContainerDb's SQL names them as before; Database::sql() puts the
prefix on). The manifest says format 3, min-reader 3: a format 2 reader
would find no tables of its names. Format 2 files, with or without an
index.sqlite entry, are read as before: the prefix is found from the
file (doc_meta).
A .qetz with an index opens from project.sqlite. Measured: the 136-folio
f2099 1.8 s, 5.2 MB (the two table sets still overlap: D3b).
Checked: tst_qetcontainer with the format 3 names, 34/34; a format 2 file
of the last commit opens from its index.sqlite and saves the same; corpus
342: 340 .qetz carry the index in project.sqlite and open from it, saved
back = eager = master on all 342. ctest 96/96 with and without
QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A .qetz now carries index.sqlite: the project database as QElectroTech
keeps it -- every table but the drawing items, which exist for built
folios only -- with each folio's own data (Diagram::headerXml()) and a
stamp of the schema it was written with (projectDataBase::indexFile(),
QETProject::writeContainerFile(): Save, --resave, --set-titleblock, a
script's and live mode's save). Written when every folio, symbol and wire
has a uuid of its own; a reader that does not know the entry skips it.
With QET_LAZY_FOLIOS=1, a .qetz with an index opens from it: the project
document is joined without its folios (QetContainer::join() with
folio_files), the database and the stores of symbol information and wire
properties are filled from the index (fillFromIndex()), each folio is made
from its own data and keeps the name of its part, read when the folio is
built (Diagram::deferredXml(), QETProject::readFolioPart(),
QetContainer::readFolio() -- pictures and engineering data back from
project.sqlite, QetContainerDb::restoreFolio()). A terminal strip finds its
terminals' folios in the database. Any other .qetz, or one whose index
cannot be used, is read whole as before.
The index is untrusted: refused unless written with this database's
schema, every table read a plain table (not a view), nothing in its schema
run (trusted_schema off); refused, the project is read whole. Checked: an
index of garbage, a wrong stamp, a table made a view -- each refused and
the project saved as a plain save.
The 136-folio f2099 as .qetz opens in 1.8 s (3.65 s read whole, 29 s
building every folio); what is left is mostly its 7 MB of symbol
definitions (0.8 s). The file grows from 2.6 to 5.2 MB, the index
overlapping project.sqlite until D3.
tst_lazyfolios: a .qetz opened from its index builds only the terminal
strip's folios, numbering read as when built, every folio shown in turn
draws and links as when built, the save as when built; fails when a folio
part's engineering data is not put back. Corpus, 342 projects: 340 .qetz
carry an index and open from it (2 have symbols sharing a uuid); saved
back as .qet = eager = master on all 342. Probe on industrial and f2099
.qetz: every folio as eager but the 3 render-noise folios of f2099. In a
real window the .qetz opens and pages building only shown folios and
partners. ctest 96/96 with and without QET_LAZY_FOLIOS=1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The project's terminal strips -- strip information, the terminals of
each physical terminal by level, bridges and what they bridge -- move to
four tables of project.sqlite, when rebuilding them from the rows gives
the same XML exactly (as QElectroTech writes them). Anything else, or
strips that carry text a save would not write, stays in project.xml.
Test: tst_qetcontainer::terminalStripsInDatabase (two strips with a
bridge and a two-level terminal on industrial.qet's own terminals: out
of project.xml, back exactly; stays in the XML when the rebuild is
broken). Outside ctest: a strip fixture saved as .qetz then .qet by
QElectroTech equals its straight resave, and the strips are rows any
SQLite tool reads.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Wga9B92BNurDEtxxtCDxf
Fourth stage of the zipped project file (#1440), format 2: what an
engineer would put in a list -- folio title-block fields and
properties, project properties, a symbol's uuid, type, prefix,
information and links, a wire's ends, number, function, colour,
section, cable and bus -- moves out of the XML into project.sqlite in
the .qetz, and lives only there. The folio files keep where things are
drawn. On opening it goes back into the document before the usual
reader runs, so nothing else in QElectroTech changes yet (the program
reading the database directly is the next step of the roadmap).
Each moved item keeps a key in the XML (qetz-key: its uuid where
unique); each moved block leaves a marker where it was, which keeps the
whitespace around it apart. A block of a shape this does not model
stays in the XML. A file is untrusted: the database is read with
trusted_schema off, query_only, and only from names that are plain
tables. Format 2 readers still open format 1; a format 1 reader
refuses format 2.
Tests (tst_qetcontainer): every example back exactly, whitespace
included; databaseHoldsTheData (industrial.qet: the folio files hold no
information, links, wire numbers or properties; the row counts match).
Outside ctest: 341 of 341 projects QET can open give the same .qet
through a format 2 .qetz.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019Wga9B92BNurDEtxxtCDxf
ispyisail