Background
Every year, the National Institute of Child Health and Human Development produces a comprehensive Annual Report, which "outlines our research endeavors and highlights our medical and scientific discoveries of the past year."
NICHD’s intramural community comprises a broad array of basic, translational, and clinical researchers. We use a range of model systems to further knowledge of developmental, molecular, and cellular biology, neurosciences, structural biology, imaging, and biophysics. Investigators working with various animal models, including fruit flies, zebrafish, rats, and mice, are supported by multiple core services, from bioinformatics and imaging, to molecular genomics. Each investigator participates in one or more team-based and future-oriented affinity groups, which build on thematic interests while staying responsive to rapidly shifting scientific priorities and new knowledge.
– Chris J. McBain, PhD, Scientific Director, NICHD, NIH
Problem
Submissions arrive by way of XML files which are exported from an internal system called DIRweb. These files have a unique syntax that's highly customized for this application, as XML files typically are.
Unfortunatlely, in previous years, the only real way to turn them into functional webpages was to painstakingly convert each and every file to HTML by hand. If a mistake is discovered on a common element of the website, such as a broken menu link, the change would have to be made on every individual HTML file. Not ideal!
As such, when work began on the 2024 edition, I proposed building a static site generator for the project.
Solution
By this time, I'd become quite comfortable with building Flask apps, and I realized that its templating engine would be perfect for converting the XML files; each tag could become its own tiny HTML template.
For instance...
<collaborators> <collaborator>John Doe</collaborator> <collaborator>Jane Doe</collaborator> </collaborators>
would be handled by...
{% if innerHTML|length > 2 %}
<div class="collaborators tag-collaborators">
<h2>Collaborators</h2>
<div class="marginify">
<ul class="usa-list nolistmargin">
{{ innerHTML|safe }}
</ul>
</div>
</div>
{% endif %}
<li class="tag-collaborator list-no-bullet">{{ innerHTML|safe }}</li>
Results
Unsurprisingly, getting just short of 100 XML files parsed correctly takes time and plenty of tweaking. As such, while it was certainly less frustrating than converting each file manually, it's difficult to judge whether time was saved this time around. However, now that all the bugs and oddities have been worked out, the 2025 edition will likely go from months of production down to weeks.
Additionally, rolling our own solution allowed for basically infinite customization. If markdown support is needed, we can just import markdown and write a few lines of code. For site navigation, we import yaml, define our menu in a YAML file, and use that to build a dynamic menu. For content that's finalized but might need some custom HTML tweaking, we have the app search for content in the locked_static folder which will override the XML parsing step.