kotschak/qelectrotech-source-mirror
0
0
Fork 0
mirror of https://github.com/qelectrotech/qelectrotech-source-mirror.git synced 2026-10-15 10:04:13 +02:00
Code Issues Actions Packages Projects Releases Wiki Activity

Open a .qetz from its database, each folio read when built (stage D2)

Browse Source 
A .qetz now carries index.sqlite: the project database as QElectroTech
keeps it -- every table but the drawing items, which exist for built
folios only -- with each folio's own data (Diagram::headerXml()) and a
stamp of the schema it was written with (projectDataBase::indexFile(),
QETProject::writeContainerFile(): Save, --resave, --set-titleblock, a
script's and live mode's save). Written when every folio, symbol and wire
has a uuid of its own; a reader that does not know the entry skips it.

With QET_LAZY_FOLIOS=1, a .qetz with an index opens from it: the project
document is joined without its folios (QetContainer::join() with
folio_files), the database and the stores of symbol information and wire
properties are filled from the index (fillFromIndex()), each folio is made
from its own data and keeps the name of its part, read when the folio is
built (Diagram::deferredXml(), QETProject::readFolioPart(),
QetContainer::readFolio() -- pictures and engineering data back from
project.sqlite, QetContainerDb::restoreFolio()). A terminal strip finds its
terminals' folios in the database. Any other .qetz, or one whose index
cannot be used, is read whole as before.

The index is untrusted: refused unless written with this database's
schema, every table read a plain table (not a view), nothing in its schema
run (trusted_schema off); refused, the project is read whole. Checked: an
index of garbage, a wrong stamp, a table made a view -- each refused and
the project saved as a plain save.

The 136-folio f2099 as .qetz opens in 1.8 s (3.65 s read whole, 29 s
building every folio); what is left is mostly its 7 MB of symbol
definitions (0.8 s). The file grows from 2.6 to 5.2 MB, the index
overlapping project.sqlite until D3.

tst_lazyfolios: a .qetz opened from its index builds only the terminal
strip's folios, numbering read as when built, every folio shown in turn
draws and links as when built, the save as when built; fails when a folio
part's engineering data is not put back. Corpus, 342 projects: 340 .qetz
carry an index and open from it (2 have symbols sharing a uuid); saved
back as .qet = eager = master on all 342. Probe on industrial and f2099
.qetz: every folio as eager but the 3 render-noise folios of f2099. In a
real window the .qetz opens and pages building only shown folios and
partners. ctest 96/96 with and without QET_LAZY_FOLIOS=1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
ispyisail
2026-10-11 18:58:35 +13:00
parent ecc7a39131
commit e89ccf2a40
15 changed files with 719 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files
sources/dataBase/projectdatabase.cpp
+278
View File
@@ -49,6 +49,8 @@
#include <QMetaEnum>
#include <QTextDocument>
#include <QFile>
#include <QCryptographicHash>
#include <QTemporaryDir>
#include <QRegularExpression>
#include <QSqlDriver>
#include <QSqlError>
@@ -865,6 +867,282 @@ bool projectDataBase::populateFromDocument(const QDomDocument &document, QString
return true;
}
/**
@brief projectDataBase::schemaStamp
@return a stamp of the tables of this database as created: an index
file (indexFile()) is read back only into a database with the same
*/
QString projectDataBase::schemaStamp() const
{
QSqlQuery query(m_data_base);
query.exec(QStringLiteral("SELECT sql FROM sqlite_master WHERE type IN ('table', 'index') "
"AND name NOT LIKE 'sqlite_%' ORDER BY name"));
QCryptographicHash hash(QCryptographicHash::Sha256);
while (query.next())
hash.addData(query.value(0).toString().toUtf8() + ';');
return QString::fromLatin1(hash.result().toHex());
}
namespace {
//The tables an index file carries, all of this database's but the
//drawing items, which exist for built folios only
QStringList indexTables(const QSqlDatabase &db)
{
QStringList tables;
QSqlQuery query(db);
query.exec(QStringLiteral("SELECT name FROM sqlite_master WHERE type = 'table' "
"AND name NOT LIKE 'sqlite_%' ORDER BY name"));
while (query.next()) {
const QString name = query.value(0).toString();
if (name != QLatin1String("shape") && name != QLatin1String("independent_text")
&& name != QLatin1String("image"))
tables << name;
}
return tables;
}
}
/**
@brief projectDataBase::indexFile
This database written to an SQLite file, for a .qetz to be opened from
(stage D2): every table but the drawing items, brought
up to date first, with the folios' own data @p folio_headers (in folio
order, each as Diagram::headerXml() writes it) and the stamp of the
schema it was written with.
@return the file, or nothing if it could not be written
*/
QByteArray projectDataBase::indexFile(const QList<std::pair<QUuid, QString>> &folio_headers)
{
if (m_content_changed) {
updateDB();
}
flushLinks();
flushElementPositions();
flushConductorProperties();
QTemporaryDir dir;
if (!dir.isValid()) {
return QByteArray();
}
const QString path = dir.filePath(QStringLiteral("index.sqlite"));
QString escaped = path;
escaped.replace(QLatin1Char('\''), QStringLiteral("''"));
QSqlQuery vacuum(m_data_base);
if (!vacuum.exec(QStringLiteral("VACUUM INTO '%1'").arg(escaped))) {
qWarning() << "projectDataBase::indexFile:" << vacuum.lastError().text();
return QByteArray();
}
const QString stamp = schemaStamp();
const QString connection = QStringLiteral("qet_index_") + QUuid::createUuid().toString();
bool ok = false;
{
QSqlDatabase file = QSqlDatabase::addDatabase(QStringLiteral("QSQLITE"), connection);
file.setDatabaseName(path);
if (file.open())
{
QSqlQuery query(file);
//Its views read the drawing items too: an index holds tables
QStringList views;
query.exec(QStringLiteral("SELECT name FROM sqlite_master WHERE type = 'view'"));
while (query.next()) views << query.value(0).toString();
for (const QString &view : std::as_const(views))
query.exec(QStringLiteral("DROP VIEW \"%1\"").arg(view));
ok = query.exec(QStringLiteral("DROP TABLE shape"))
&& query.exec(QStringLiteral("DROP TABLE independent_text"))
&& query.exec(QStringLiteral("DROP TABLE image"))
&& query.exec(QStringLiteral("CREATE TABLE index_meta (key TEXT PRIMARY KEY, value TEXT)"))
&& query.exec(QStringLiteral("CREATE TABLE folio_header (pos INTEGER PRIMARY KEY, "
"uuid TEXT NOT NULL, xml TEXT NOT NULL)"));
file.transaction();
query.prepare(QStringLiteral("INSERT INTO index_meta (key, value) VALUES (?, ?)"));
for (const auto &meta : {std::pair<QString, QString>{QStringLiteral("schema"), stamp},
std::pair<QString, QString>{QStringLiteral("writer"),
QStringLiteral("QElectroTech ") + QetVersion::currentVersion().toString()}}) {
query.addBindValue(meta.first);
query.addBindValue(meta.second);
ok = ok && query.exec();
}
query.prepare(QStringLiteral("INSERT INTO folio_header (pos, uuid, xml) VALUES (?, ?, ?)"));
int pos = 0;
for (const auto &header : folio_headers) {
query.addBindValue(pos++);
query.addBindValue(header.first.toString());
query.addBindValue(header.second);
ok = ok && query.exec();
}
ok = file.commit() && ok;
ok = query.exec(QStringLiteral("VACUUM")) && ok;
file.close();
}
}
QSqlDatabase::removeDatabase(connection);
if (!ok) {
return QByteArray();
}
QFile file(path);
if (!file.open(QIODevice::ReadOnly)) {
return QByteArray();
}
return file.readAll();
}
/**
@brief projectDataBase::discardFill
Undo fillFromIndex(): the project is read from its document instead
*/
void projectDataBase::discardFill()
{
QSqlQuery query(m_data_base);
for (const QString &table : indexTables(m_data_base))
query.exec(QStringLiteral("DELETE FROM \"%1\"").arg(table));
m_element_information.clear();
m_folio_titleblocks.clear();
m_conductor_properties.clear();
m_conductor_attributes.clear();
m_prefilled_elements.clear();
m_prefilled_folios.clear();
m_prefilled_conductors.clear();
m_rows_from_document = false;
m_content_changed = true;
}
/**
@brief projectDataBase::fillFromIndex
Fill this database from @p index, a file indexFile() wrote, instead of
from the project's document: its tables, and the stores of symbol
information and wire properties (the folios' title blocks are filled
from @p folio_headers by the folios). The file is untrusted: read only
when written with this database's schema, its tables plain tables, and
nothing in its schema run.
@param folio_headers : the folios' own data, in folio order
@return false, with @p why set and nothing changed in the tables, if
the file cannot be read
*/
bool projectDataBase::fillFromIndex(const QByteArray &index,
QList<std::pair<QUuid, QString>> *folio_headers,
QString *why)
{
auto refuse = [why](const QString &reason) {
if (why) *why = reason;
return false;
};
QTemporaryDir dir;
if (!dir.isValid()) {
return refuse(QStringLiteral("no temporary folder"));
}
const QString path = dir.filePath(QStringLiteral("index.sqlite"));
{
QFile file(path);
if (!file.open(QIODevice::WriteOnly) || file.write(index) != index.size()) {
return refuse(QStringLiteral("cannot write the index to a temporary file"));
}
}
QString escaped = path;
escaped.replace(QLatin1Char('\''), QStringLiteral("''"));
QSqlQuery query(m_data_base);
query.exec(QStringLiteral("PRAGMA trusted_schema = OFF"));
if (!query.exec(QStringLiteral("ATTACH DATABASE '%1' AS idx").arg(escaped))) {
return refuse(QStringLiteral("the index cannot be read"));
}
auto detach = [this]() {
QSqlQuery q(m_data_base);
q.exec(QStringLiteral("DETACH DATABASE idx"));
};
const QStringList tables = indexTables(m_data_base);
auto isTable = [this](const QString &name) {
QSqlQuery kind(m_data_base);
kind.prepare(QStringLiteral("SELECT type FROM idx.sqlite_master WHERE name = ?"));
kind.addBindValue(name);
return kind.exec() && kind.next() && kind.value(0).toString() == QLatin1String("table");
};
for (const QString &table : tables + QStringList{QStringLiteral("index_meta"),
QStringLiteral("folio_header")}) {
if (!isTable(table)) {
detach();
return refuse(QStringLiteral("the index has no table %1").arg(table));
}
}
query.exec(QStringLiteral("SELECT value FROM idx.index_meta WHERE key = 'schema'"));
if (!query.next() || query.value(0).toString() != schemaStamp()) {
detach();
return refuse(QStringLiteral("the index was written with another database schema"));
}
m_data_base.transaction();
bool ok = true;
for (const QString &table : tables) {
QSqlQuery columns(m_data_base);
columns.exec(QStringLiteral("PRAGMA main.table_info(\"%1\")").arg(table));
QStringList names;
while (columns.next())
names << QStringLiteral("\"%1\"").arg(columns.value(1).toString());
const QString list = names.join(QLatin1Char(','));
ok = ok && query.exec(QStringLiteral("DELETE FROM main.\"%1\"").arg(table))
&& query.exec(QStringLiteral("INSERT INTO main.\"%1\" (%2) SELECT %2 FROM idx.\"%1\"")
.arg(table, list));
}
if (!ok) {
m_data_base.rollback();
detach();
return refuse(QStringLiteral("the index's rows cannot be read"));
}
m_data_base.commit();
folio_headers->clear();
query.exec(QStringLiteral("SELECT uuid, xml FROM idx.folio_header ORDER BY pos"));
while (query.next())
folio_headers->append({QUuid(query.value(0).toString()), query.value(1).toString()});
detach();
//The stores, from their tables, as the document would fill them
m_prefilled_elements.clear();
m_prefilled_folios.clear();
m_prefilled_conductors.clear();
for (PrefillTally &tally : m_prefill_tally) tally = PrefillTally();
query.exec(QStringLiteral("SELECT element_uuid, name, value, show FROM element_information "
"ORDER BY element_uuid, ord"));
while (query.next()) {
const QUuid uuid(query.value(0).toString());
m_prefilled_elements[uuid].addValue(query.value(1).toString(), query.value(2).toString(),
query.value(3).toInt() != 0);
}
QHash<QUuid, QHash<QString, QString>> attributes;
query.exec(QStringLiteral("SELECT conductor_uuid, name, value FROM conductor_properties"));
while (query.next())
attributes[QUuid(query.value(0).toString())].insert(query.value(1).toString(),
query.value(2).toString());
QDomDocument scratch;
for (auto it = attributes.cbegin() ; it != attributes.cend() ; ++it) {
QDomElement e = scratch.createElement(QStringLiteral("conductor"));
for (auto a = it.value().cbegin() ; a != it.value().cend() ; ++a)
e.setAttribute(a.key(), a.value());
ConductorProperties properties;
properties.fromXml(e);
m_prefilled_conductors.insert(it.key(), properties);
}
for (auto it = m_prefilled_elements.cbegin() ; it != m_prefilled_elements.cend() ; ++it)
m_element_information.insert(it.key(), it.value());
for (auto it = m_prefilled_conductors.cbegin() ; it != m_prefilled_conductors.cend() ; ++it)
m_conductor_properties.insert(it.key(), it.value());
m_conductor_attributes = attributes;
for (const auto &header : std::as_const(*folio_headers)) {
QDomDocument document;
if (!document.setContent(header.second)) continue;
TitleBlockProperties properties;
properties.fromXml(document.documentElement());
properties.version = QetVersion::displayedVersion();
properties.collection = QET::QetCollection::Embedded;
m_prefilled_folios.insert(header.first, properties);
m_folio_titleblocks.insert(header.first, properties);
}
m_rows_from_document = true;
m_content_changed = false;
emit dataBaseUpdated();
return true;
}
/**
@brief projectDataBase::foliosWithUnjoinedWires
@return the uuids of the folios whose document has a wire that building