# type

id: blog
directory: _posts
list_fields: ["title"]
default_format: md
markdown_body_is_field: content
json_schema:
{"type": "object", "properties": {"title": {"type": "string", "title": "Title", "description": ""}, "description": {"type": "string", "title": "Description", "description": ""}, "content": {"type": "string", "title": "Content", "description": "", "format": "textarea"}, "layout": {"type": "string", "title": "layout", "description": "This should always be set to blog"}}, "$schema": "http://json-schema.org/draft-07/schema"}


# record_blog

id: 2022-03-31-python-library-version-0-3-1-released
git_filename: _posts/2022-03-31-python-library-version-0-3-1-released.md
format: md
field_title: Python Library Version 0.3.1 Released
field_layout: blog
data:
{"layout": "blog", "title": "Python Library Version 0.3.1 Released", "content": "We released Version 0.3.1 of our Python Library.\n\nFor details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#031---2022-03-31)"}

field_content:
We released Version 0.3.1 of our Python Library.

For details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#031---2022-03-31)


id: 2021-07-22-python-library-version-0-2-0-released
git_filename: _posts/2021-07-22-python-library-version-0-2-0-released.md
format: md
field_title: Python Library Version 0.2.0 Released
field_layout: blog
data:
{"layout": "blog", "title": "Python Library Version 0.2.0 Released", "content": "We released Version 0.2.0 of our Python Library.\n\nFor details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#020---2021-07-22)"}

field_content:
We released Version 0.2.0 of our Python Library.

For details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#020---2021-07-22)


id: 2024-07-12-python-library-version-0-6-0-released
git_filename: _posts/2024-07-12-python-library-version-0-6-0-released.md
format: md
field_title: Python Library Version 0.6.0 Released
field_layout: blog
data:
{"layout": "blog", "title": "Python Library Version 0.6.0 Released", "content": "We released Version 0.6.0 of our Python Library.\n\nFor details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#060---2024-07-12)"}

field_content:
We released Version 0.6.0 of our Python Library.

For details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#060---2024-07-12)


id: 2022-05-05-python-library-version-0-4-0-released
git_filename: _posts/2022-05-05-python-library-version-0-4-0-released.md
format: md
field_title: Python Library Version 0.4.0 Released
field_layout: blog
data:
{"layout": "blog", "title": "Python Library Version 0.4.0 Released", "content": "We released Version 0.4.0 of our Python Library.\n\nFor details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#040---2022-05-05)"}

field_content:
We released Version 0.4.0 of our Python Library.

For details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#040---2022-05-05)


id: 2024-11-06-python-library-version-0-7-0-released
git_filename: _posts/2024-11-06-python-library-version-0-7-0-released.md
format: md
field_title: Python Library Version 0.7.0 Released
field_layout: blog
data:
{"layout": "blog", "title": "Python Library Version 0.7.0 Released", "content": "We released Version 0.7.0 of our Python Library.\n\nFor details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#070---2024-11-06)"}

field_content:
We released Version 0.7.0 of our Python Library.

For details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#070---2024-11-06)


id: 2022-02-26-work-continues
git_filename: _posts/2022-02-26-work-continues.md
format: md
field_title: Work Continues
field_layout: blog
data:
{"layout": "blog", "title": "Work Continues", "content": "We wrote up an introduction to the the project and some notes on recent work on [a Hashnode blog](https://odscjames.hashnode.dev/datatig-helps-you-crowd-source-data-in-a-git-repository)."}

field_content:
We wrote up an introduction to the the project and some notes on recent work on [a Hashnode blog](https://odscjames.hashnode.dev/datatig-helps-you-crowd-source-data-in-a-git-repository).


id: 2023-01-03-python-library-version-0-5-0-released
git_filename: _posts/2023-01-03-python-library-version-0-5-0-released.md
format: md
field_title: Python Library Version 0.5.0 Released
field_layout: blog
data:
{"layout": "blog", "title": "Python Library Version 0.5.0 Released", "content": "We released Version 0.5.0 of our Python Library.\n\nFor details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#050---2023-01-03)"}

field_content:
We released Version 0.5.0 of our Python Library.

For details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#050---2023-01-03)


id: 2024-12-24-talk
git_filename: _posts/2024-12-24-talk.md
format: md
field_title: Talk about this project at local Python event
field_layout: blog
data:
{"layout": "blog", "title": "Talk about this project at local Python event", "content": "James was meant to give a talk at a local Python event in September. \nUnfortunately at the last minute the sponsor, venue and event fell through. \nInstead of wasting the talk we'll post it here (as we were going to do afterwards anyway). \nHopefully it's not our fault the event had problems!\n\n----------------------------\n\n![slide saying hello](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-01.png)\n\n\nHello, I want to start by showing you some websites.\n\n----------------------------\n\n![slide of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-02.png)\n\n[Endoflife.date](https://endoflife.date/) is a website that lists the dates support for different versions of software packages ends, in a consistent interface. \nIt's great for quickly looking up dates at an easily known URL.\n\nThey track over 5000 releases from over 300 products.\n\n----------------------------\n\n![slide of screenshot of Python proposal](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-03.png)\n\n\nBTW do you know Python is thinking of moving to calendar versioning? So the next version will be 3.25. \nThis makes it easy to work out which year support for this version ends.\n\n----------------------------\n\n\n![slide of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-04.png)\n\n\nUbuntu has been on calendar versioning for a while, but personally I still find it easier to look up dates here.\n\nAlso, this website is great for looking up which codename is for which version. \nI never bothered to memorize them and this way when someone says \"Are you using Ubuntu Zonky Zebra?\" I can just look up what number that means here. \n\n----------------------------\n\n\n![slide  of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-05.png)\n\n[Open Data Scotland](https://opendata.scot/) lists open data sets from across Scotland.\n\n\n----------------------------\n\n\n![slide  of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-06.png)\n\nIt has information about over 2000 data sets and the different formats each is available in.\n\n----------------------------\n\n\n![slide of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-07.png)\n\n[Jam stack](https://jamstack.org/generators/) lists static website generators - 55 written in Python.\n\nWhy am I showing you these websites? They all crowdsource their data in Git repositories.\n\n----------------------------\n\n\n\n![slide  of screenshot of git repository](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-08.png)\n\n[Here's the jam stack repository](https://github.com/jamstack/jamstack.org/blob/main/src/site/generators/sphinx.md?plain=1). \nYou can see each product has one file, and each file has markdown content about that product. The YAML block at the top contains information such as the programming language.\n\n\n----------------------------\n\n\n![slide of screenshot of git repository](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-09.png)\n\n\n[This is the repository for Open Data Scotland](https://github.com/OpenDataScotland/jkan/blob/gh-pages/_datasets/city%2Bof%2Bedinburgh%2Bcouncil-polling%2Bplaces%2B2024.md?plain=1). \nA file for each data set and the YAML block lists the different formats.\n\n----------------------------\n\n\n![slide of screenshot of git repository](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-10.png)\n\n[This is the repository for endoflife website](https://github.com/endoflife-date/endoflife.date/blob/master/products/python.md?plain=1).\n\n\n----------------------------\n\n\n![slide of screenshot of git repository](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-11.png)\n\nBy the way, there will be a lot of GitHub screenshots because GitHub is the most popular git host. \nBut other Git hosts are available and most of the things I will say apply to all Git hosts. \n[This is a project hosted on Codeberg](https://codeberg.org/swiso/website/src/branch/main).\n\n----------------------------\n\n\n\n![slide of diagram, described below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-12.png)\n\nSo basically what's happening here is people are contributing data into a Git repository, \nand from this they then publish a static website.\n\nIs this a good idea?\n\n----------------------------\n\n\n![slide of pros, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-13.png)\n\nPros:\n\nGood safe storage and hosting of data - Git has been around for years and is rock solid.\n\nHistory - you get a history of the data for free.\n\nData in a kinda open format for reuse - the data is just in files and there are many ways to process it.\n\nLots of tools for building a nice static website with the data  - I just showed you a list of 361 tools.\n\nPeople can contribute with PR's, an approvals processs & diffs - and they already know how to use Git.\n\nHave one file per record for easy Git merges with few Git conflicts (more on this later).\n\nLots of no money hosting options with no sysadmin work - \nyou can host a public repository on GitHub for free and use Actions to host a website on Pages for free. \nNo sysadmin worries or pager duties either.\n\n----------------------------\n\n\n![slide of pros, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-14.png)\n\nAnother advantage is special one-off data tasks.\n\nWith a traditional web app their may be problems. Your app may be set up nicely to allow editing one record at a time.\n\nBut what happens if someone comes along and says they need to add a record for most countries - 250 records - at once? There may be no nice way to do this and it may be difficult.\n\nThis was a real request on one of our projects.\n\n----------------------------\n\n\n\n\n![slide of diagram, described below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-15.png)\n\n\nWith a Git repository, if the existing tools don't support it that's not a problem. The person can write a one off script to generate the files and they can all be put in one PR to be checked.\n\n----------------------------\n\n\n![slide of cons, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-16.png)\n\nCons:\n\nHistory can be a con as well as a pro. Some projects might have the requirement that data can be fully deleted from them on request, including from the history. While rewriting history is possible in Git, it's not easy.\n\n----------------------------\n\n\n![slide of cons, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-17.png)\n\nAnd one really big disadvantage ...\n\nEarlier I said people are familiar with Git, but that's not true. \nOnly computer programmers use it, and some of them don't like it. \nThis creates a high technical barrier to contributing data \nand gives me a chance to bring out [one of my favorite comics about tech accessibility](https://howfuckedismydatabase.com/nosql/):\n\nBob: So how do I query the database?\n\nJim: It's not a database. It's a key-value store!\n\nBob: Ok, It's not a database. How do I query it?\n\nJim: You write a distributed map reduce function in Erlang!\n\nBob: Did you just tell me to go f*** myself?\n\nJim: I believe I did, Bob.\n\n----------------------------\n\n\n![slide  of screenshet of iCal file](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-18.png)\n\nI ran [Open Tech Calendar](https://opentechcalendar.co.uk/), listing tech events around the UK for years.\n\nAt one point someone suggested to me that it should just be a repository of iCal files that people could edit by hand. \n(iCal files are the standard for calendar data.)\n\nSo firstly that's a barrier to contributing as Git knowledge is required. \nInstead we had a nice form on our website and many non-programmers contributed events over the years.\n\nSecondly that's a barrier to contributing data as the iCal format is really annoying to edit by hand! \nWould you enjoy hand editing this? Do you think you could get it right first go?!\n\nI like to think part of the success of Open Tech Calendar was that I completely ignored that advice.\n\n----------------------------\n\n\n\n![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-19.png)\n\nSo should you use a Git repository to crowdsource data?\n\nIn some situations the technical barrier problem is fine as all the people \ncontributing to the project are probably technical anyway (ie endoflife.date).\n\nTo try and reduce the technical barrier, many projects accept contributions via issues.\nHowever, this is still a barrier as people need to open an account on a website they are not familiar with \nand work out the new issue form - which is almost like a forum post but with extra weird options like \"assigned to\".\n\nSo is this a good idea?\n\n----------------------------\n\n\n![slide of screenshots](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-20.png)\n\nWell ... everyone does this anyway!\n\n[Org-id](https://org-id.guide/) is a project we run at my work. \nIt lists places that list companies. \nTo explain why this is important to our work isn't relevant now and would take too long \n(but [this blog post explains](https://medium.com/opendatacoop/why-do-open-organisational-identifiers-matter-46af05ab30a) if you are interested).\n\n[Up for grabs](https://up-for-grabs.net) lists projects with issues marked as suitable for newcomers, to encourage people into open source.\n\n[This website lists LLM engines and how open they are](https://opening-up-chatgpt.github.io/), \nby various criteria.\n\n[Switching software](https://switching.software/) lists alternatives to proprietary programs.\n\n\n----------------------------\n\n\n\n![slide  of screenshots](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-21.png)\n\n\nI could go on - but you get the point.\n\n(Projects: \n[1](https://areweanticheatyet.com/)\n[2](https://fediverseexplorations.org/)\n[3](https://github.com/ralexander-phi/relicensing-monitor)\n[4](https://getindie.wiki/))\n\n----------------------------\n\n\n![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-22.png)\n\nProblem / Opportunity: Everyone reinvents the wheel!\n\nSome static site generators are very popular, but everyone tends to reinvent the wheel when it comes to data tools and workflows.\n\n----------------------------\n\n\n![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-23.png)\n\nSo this explains our project, DataTig.\n\nIt's for when a community of people want to crowdsource a data set and they use a Git repository to store the data. \n\n* Lots of people do this & reinvent wheel each time they do\n* Can shared tools help these projects?\n* Help people get and reuse data\n* Help people check data\n* Help people contribute data\n* (All Open Source)\n\n----------------------------\n\n\n\n![slide of screenshot](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-24.png)\n\n\nSo if you have a repository with data in it ...\n\n----------------------------\n\n\n![slide of screenshot](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-25.png)\n\n[You add a config file](https://github.com/org-id/register/blob/main/datatig.yaml). \nThis mainly describes the format of the data and the fields in it.\n\n----------------------------\n\n\n\n![slide of screenshot](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-26.png)\n\n\nDataTig can then give you lots of advantages.\n\nA search and filter UI to find records.\n\n----------------------------\n\n\n![slide of screenshot](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-27.png)\n\nSee the details of any records\n\n----------------------------\n\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-28.png)\n\n\nFor reusing the data, you can export it all as a SQLite file and run custom queries against it.\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-29.png)\n\nYou can export it as spreadsheets in [Frictionless Data format](https://frictionlessdata.io/).\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-30.png)\n\n\nThere's a JSON API for other apps to use.\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-31.png)\n\n\nFor helping the quality of the data it can check for errors and highlight them for you.\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-32.png)\n\nWe can check that any links in the data are valid.\n\n----------------------------\n\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-33.png)\n\n\nWe can show you information about the fields.\n\nHere you can see some tags use underscores and some dashes (third_party vs third-party) - this project should standardise on one.\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-34.png)\n\nFor people contributing data, we can make web forms. \n\nThis way we hope to try and make it easier for people and overcome the technical barriers to contributing.\n\n----------------------------\n\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-35.png)\n\nFor Markdown files, this can include a nice editor for the Markdown body and the YAML tags at the top.\n\n----------------------------\n\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-36.png)\n\n\nThis is an experimental UI, showing the differences in data between a branch and the main branch. We think this will be useful attached to PR requests.\n\n----------------------------\n\n\n![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-37.png)\n\nWays to use this.\n\n----------------------------\n\n\n![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-38.png)\n\n\nYou can use it as [a Python library](https://pypi.org/project/DataTig/). It can produce a static website, or the SQLite or spreadsheet exports you saw earlier.\n\nIt can also be used in C.I. pipelines to check the quality of data.\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-39-and-42.png)\n\nEvery time you push up a commit to GitHub, you can have it run tests and report any problems in the data.\n\n----------------------------\n\n\n\n![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-40.png)\n\n\nThis is really easy to do in Python. If you have files in a Git repository, you can write a Python script to check them. It can be anything you can check in Python.\n\nFor example, if you have images you can check they are a minimum size.\n\n----------------------------\n\n![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-41.png)\n\n\nIf the script finds no problems, it exits normally.\n\nBut if the script finds problems, it exits by calling `sys.exit(-1)`. (An appropriate import statement is needed too )\n\nThen you just need a really simple GitHub Action config file.\n\n* Check out the repository\n* Install python\n* Install your python script from wherever it is\n* Run your python script.\n\nThat's it!\n\n(We have [docs on this](https://datatig.readthedocs.io/en/latest/how-to-guide/use-github-actions-to-check-your-data.html).)\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-39-and-42.png)\n\nYou'll now get a check every time a commit is pushed up.\n\nYou can require these to pass before people are allowed to merge PRs.\n\n----------------------------\n\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-43.png)\n\nWe also have [our Hub web application](https://hub.datatig.com/) available. This is a Django application that uses the Python Library and can host multiple repositories and multiple branches.\n\nIf you log in with GitHub you can fill out a form, and sometimes it will try to make the commits in a new branch in GitHub for you. This hopefully reduces the technical barrier to contributing.\n\nIt does a few other things you can't do in a static website, but these days you can do so much with a static website and client side JavaScript and WASM in the web browser.\n\n----------------------------\n\n\n\n![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-44.png)\n\n\nThere is also an experimental local server which you can run directly in your machine. When you fill out a form in your web browser, it will save the data on your disk for you to commit as normal.\n\n----------------------------\n\n\n![slide of heading, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-45.png)\n\n\nIs this also a barrier to entry? Some of these screens are very data heavy.\n\n----------------------------\n\n\n![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-46.png)\n\nBut you can use as much or as little of this as you want.\n\nYou can use GitHub Actions to do one build from one repository and create both a custom static website with you own messaging and a DataTig static website in a subdirectory for data users and contributors.\n\n(We have [docs on this](https://datatig.readthedocs.io/en/latest/how-to-guide/use-github-actions-to-build-and-host-your-site.html).)\n\n\n----------------------------\n\n\n![slide of heading, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-47.png)\n\nHow to store the data in the git repository?\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-48.png)\n\n\nOne way of storing data is a separate file for each record and a YAML block and a markdown body. This is a pretty good way to organise the data in the repository.\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-49.png)\n\nYou can also just have YAML or JSON files. We think JSON files are harder to read and hand edit, and the trailing slashes makes for messier diffs.\n\n\n----------------------------\n\n\n![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-50.png)\n\nBut the main thing to avoid is putting many records in one file. We've often seen one JSON file with a big list.\n\nThis causes problems because as many people edit different records at the same time, the chances of messy merges and maybe even merge conflicts increases. The history gets messier too.\n\nWe've sometimes seen a CSV file. This is hard to edit and operations like adding a field are harder to do.\n\n\n----------------------------\n\n\n![slide of information seen before](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-51-and-53.png)\n\n\nSo that's our project.\n\n----------------------------\n\n\n![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-52.png)\n\nBut other ways of running community data projects are available, and we've given you one big reason why you might not want to use Git.\n\nHowever you choose to run your project, we think it's important to think about the data aspects of it. \n\n* Will you ask people to send you data?\n* Can you reuse existing data?\n* How can you keep the data up to date?\n* How can you encourage others to use your data?\n* How can you encourage others to REuse your data?\n* Which existing data standards should you use to publish your data?\n* How can you spread the load of editing?\n* How can you test and demonstrate the quality of your data?\n* Are other platforms already suitable (eg Wikidata, Open Street Map)?\n* How can you tie this all together?\n\n\n[We wrote this blog post that goes into that more](http://web.archive.org/web/20240621164805/https://opendataservices.coop/blog/2020/05/07/community-data.html).\n\n----------------------------\n\n\n![slide of information seen before](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-51-and-53.png)\n\n\nThank you!\n\n[Find out more](https://www.datatig.com)"}

field_content:
James was meant to give a talk at a local Python event in September. 
Unfortunately at the last minute the sponsor, venue and event fell through. 
Instead of wasting the talk we'll post it here (as we were going to do afterwards anyway). 
Hopefully it's not our fault the event had problems!

----------------------------

![slide saying hello](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-01.png)


Hello, I want to start by showing you some websites.

----------------------------

![slide of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-02.png)

[Endoflife.date](https://endoflife.date/) is a website that lists the dates support for different versions of software packages ends, in a consistent interface. 
It's great for quickly looking up dates at an easily known URL.

They track over 5000 releases from over 300 products.

----------------------------

![slide of screenshot of Python proposal](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-03.png)


BTW do you know Python is thinking of moving to calendar versioning? So the next version will be 3.25. 
This makes it easy to work out which year support for this version ends.

----------------------------


![slide of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-04.png)


Ubuntu has been on calendar versioning for a while, but personally I still find it easier to look up dates here.

Also, this website is great for looking up which codename is for which version. 
I never bothered to memorize them and this way when someone says "Are you using Ubuntu Zonky Zebra?" I can just look up what number that means here. 

----------------------------


![slide  of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-05.png)

[Open Data Scotland](https://opendata.scot/) lists open data sets from across Scotland.


----------------------------


![slide  of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-06.png)

It has information about over 2000 data sets and the different formats each is available in.

----------------------------


![slide of screenshot of website](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-07.png)

[Jam stack](https://jamstack.org/generators/) lists static website generators - 55 written in Python.

Why am I showing you these websites? They all crowdsource their data in Git repositories.

----------------------------



![slide  of screenshot of git repository](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-08.png)

[Here's the jam stack repository](https://github.com/jamstack/jamstack.org/blob/main/src/site/generators/sphinx.md?plain=1). 
You can see each product has one file, and each file has markdown content about that product. The YAML block at the top contains information such as the programming language.


----------------------------


![slide of screenshot of git repository](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-09.png)


[This is the repository for Open Data Scotland](https://github.com/OpenDataScotland/jkan/blob/gh-pages/_datasets/city%2Bof%2Bedinburgh%2Bcouncil-polling%2Bplaces%2B2024.md?plain=1). 
A file for each data set and the YAML block lists the different formats.

----------------------------


![slide of screenshot of git repository](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-10.png)

[This is the repository for endoflife website](https://github.com/endoflife-date/endoflife.date/blob/master/products/python.md?plain=1).


----------------------------


![slide of screenshot of git repository](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-11.png)

By the way, there will be a lot of GitHub screenshots because GitHub is the most popular git host. 
But other Git hosts are available and most of the things I will say apply to all Git hosts. 
[This is a project hosted on Codeberg](https://codeberg.org/swiso/website/src/branch/main).

----------------------------



![slide of diagram, described below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-12.png)

So basically what's happening here is people are contributing data into a Git repository, 
and from this they then publish a static website.

Is this a good idea?

----------------------------


![slide of pros, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-13.png)

Pros:

Good safe storage and hosting of data - Git has been around for years and is rock solid.

History - you get a history of the data for free.

Data in a kinda open format for reuse - the data is just in files and there are many ways to process it.

Lots of tools for building a nice static website with the data  - I just showed you a list of 361 tools.

People can contribute with PR's, an approvals processs & diffs - and they already know how to use Git.

Have one file per record for easy Git merges with few Git conflicts (more on this later).

Lots of no money hosting options with no sysadmin work - 
you can host a public repository on GitHub for free and use Actions to host a website on Pages for free. 
No sysadmin worries or pager duties either.

----------------------------


![slide of pros, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-14.png)

Another advantage is special one-off data tasks.

With a traditional web app their may be problems. Your app may be set up nicely to allow editing one record at a time.

But what happens if someone comes along and says they need to add a record for most countries - 250 records - at once? There may be no nice way to do this and it may be difficult.

This was a real request on one of our projects.

----------------------------




![slide of diagram, described below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-15.png)


With a Git repository, if the existing tools don't support it that's not a problem. The person can write a one off script to generate the files and they can all be put in one PR to be checked.

----------------------------


![slide of cons, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-16.png)

Cons:

History can be a con as well as a pro. Some projects might have the requirement that data can be fully deleted from them on request, including from the history. While rewriting history is possible in Git, it's not easy.

----------------------------


![slide of cons, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-17.png)

And one really big disadvantage ...

Earlier I said people are familiar with Git, but that's not true. 
Only computer programmers use it, and some of them don't like it. 
This creates a high technical barrier to contributing data 
and gives me a chance to bring out [one of my favorite comics about tech accessibility](https://howfuckedismydatabase.com/nosql/):

Bob: So how do I query the database?

Jim: It's not a database. It's a key-value store!

Bob: Ok, It's not a database. How do I query it?

Jim: You write a distributed map reduce function in Erlang!

Bob: Did you just tell me to go f*** myself?

Jim: I believe I did, Bob.

----------------------------


![slide  of screenshet of iCal file](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-18.png)

I ran [Open Tech Calendar](https://opentechcalendar.co.uk/), listing tech events around the UK for years.

At one point someone suggested to me that it should just be a repository of iCal files that people could edit by hand. 
(iCal files are the standard for calendar data.)

So firstly that's a barrier to contributing as Git knowledge is required. 
Instead we had a nice form on our website and many non-programmers contributed events over the years.

Secondly that's a barrier to contributing data as the iCal format is really annoying to edit by hand! 
Would you enjoy hand editing this? Do you think you could get it right first go?!

I like to think part of the success of Open Tech Calendar was that I completely ignored that advice.

----------------------------



![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-19.png)

So should you use a Git repository to crowdsource data?

In some situations the technical barrier problem is fine as all the people 
contributing to the project are probably technical anyway (ie endoflife.date).

To try and reduce the technical barrier, many projects accept contributions via issues.
However, this is still a barrier as people need to open an account on a website they are not familiar with 
and work out the new issue form - which is almost like a forum post but with extra weird options like "assigned to".

So is this a good idea?

----------------------------


![slide of screenshots](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-20.png)

Well ... everyone does this anyway!

[Org-id](https://org-id.guide/) is a project we run at my work. 
It lists places that list companies. 
To explain why this is important to our work isn't relevant now and would take too long 
(but [this blog post explains](https://medium.com/opendatacoop/why-do-open-organisational-identifiers-matter-46af05ab30a) if you are interested).

[Up for grabs](https://up-for-grabs.net) lists projects with issues marked as suitable for newcomers, to encourage people into open source.

[This website lists LLM engines and how open they are](https://opening-up-chatgpt.github.io/), 
by various criteria.

[Switching software](https://switching.software/) lists alternatives to proprietary programs.


----------------------------



![slide  of screenshots](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-21.png)


I could go on - but you get the point.

(Projects: 
[1](https://areweanticheatyet.com/)
[2](https://fediverseexplorations.org/)
[3](https://github.com/ralexander-phi/relicensing-monitor)
[4](https://getindie.wiki/))

----------------------------


![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-22.png)

Problem / Opportunity: Everyone reinvents the wheel!

Some static site generators are very popular, but everyone tends to reinvent the wheel when it comes to data tools and workflows.

----------------------------


![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-23.png)

So this explains our project, DataTig.

It's for when a community of people want to crowdsource a data set and they use a Git repository to store the data. 

* Lots of people do this & reinvent wheel each time they do
* Can shared tools help these projects?
* Help people get and reuse data
* Help people check data
* Help people contribute data
* (All Open Source)

----------------------------



![slide of screenshot](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-24.png)


So if you have a repository with data in it ...

----------------------------


![slide of screenshot](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-25.png)

[You add a config file](https://github.com/org-id/register/blob/main/datatig.yaml). 
This mainly describes the format of the data and the fields in it.

----------------------------



![slide of screenshot](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-26.png)


DataTig can then give you lots of advantages.

A search and filter UI to find records.

----------------------------


![slide of screenshot](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-27.png)

See the details of any records

----------------------------



![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-28.png)


For reusing the data, you can export it all as a SQLite file and run custom queries against it.

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-29.png)

You can export it as spreadsheets in [Frictionless Data format](https://frictionlessdata.io/).

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-30.png)


There's a JSON API for other apps to use.

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-31.png)


For helping the quality of the data it can check for errors and highlight them for you.

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-32.png)

We can check that any links in the data are valid.

----------------------------



![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-33.png)


We can show you information about the fields.

Here you can see some tags use underscores and some dashes (third_party vs third-party) - this project should standardise on one.

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-34.png)

For people contributing data, we can make web forms. 

This way we hope to try and make it easier for people and overcome the technical barriers to contributing.

----------------------------



![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-35.png)

For Markdown files, this can include a nice editor for the Markdown body and the YAML tags at the top.

----------------------------



![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-36.png)


This is an experimental UI, showing the differences in data between a branch and the main branch. We think this will be useful attached to PR requests.

----------------------------


![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-37.png)

Ways to use this.

----------------------------


![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-38.png)


You can use it as [a Python library](https://pypi.org/project/DataTig/). It can produce a static website, or the SQLite or spreadsheet exports you saw earlier.

It can also be used in C.I. pipelines to check the quality of data.

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-39-and-42.png)

Every time you push up a commit to GitHub, you can have it run tests and report any problems in the data.

----------------------------



![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-40.png)


This is really easy to do in Python. If you have files in a Git repository, you can write a Python script to check them. It can be anything you can check in Python.

For example, if you have images you can check they are a minimum size.

----------------------------

![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-41.png)


If the script finds no problems, it exits normally.

But if the script finds problems, it exits by calling `sys.exit(-1)`. (An appropriate import statement is needed too )

Then you just need a really simple GitHub Action config file.

* Check out the repository
* Install python
* Install your python script from wherever it is
* Run your python script.

That's it!

(We have [docs on this](https://datatig.readthedocs.io/en/latest/how-to-guide/use-github-actions-to-check-your-data.html).)

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-39-and-42.png)

You'll now get a check every time a commit is pushed up.

You can require these to pass before people are allowed to merge PRs.

----------------------------



![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-43.png)

We also have [our Hub web application](https://hub.datatig.com/) available. This is a Django application that uses the Python Library and can host multiple repositories and multiple branches.

If you log in with GitHub you can fill out a form, and sometimes it will try to make the commits in a new branch in GitHub for you. This hopefully reduces the technical barrier to contributing.

It does a few other things you can't do in a static website, but these days you can do so much with a static website and client side JavaScript and WASM in the web browser.

----------------------------



![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-44.png)


There is also an experimental local server which you can run directly in your machine. When you fill out a form in your web browser, it will save the data on your disk for you to commit as normal.

----------------------------


![slide of heading, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-45.png)


Is this also a barrier to entry? Some of these screens are very data heavy.

----------------------------


![slide of diagram, explained below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-46.png)

But you can use as much or as little of this as you want.

You can use GitHub Actions to do one build from one repository and create both a custom static website with you own messaging and a DataTig static website in a subdirectory for data users and contributors.

(We have [docs on this](https://datatig.readthedocs.io/en/latest/how-to-guide/use-github-actions-to-build-and-host-your-site.html).)


----------------------------


![slide of heading, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-47.png)

How to store the data in the git repository?

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-48.png)


One way of storing data is a separate file for each record and a YAML block and a markdown body. This is a pretty good way to organise the data in the repository.

----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-49.png)

You can also just have YAML or JSON files. We think JSON files are harder to read and hand edit, and the trailing slashes makes for messier diffs.


----------------------------


![slide of screenshot ](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-50.png)

But the main thing to avoid is putting many records in one file. We've often seen one JSON file with a big list.

This causes problems because as many people edit different records at the same time, the chances of messy merges and maybe even merge conflicts increases. The history gets messier too.

We've sometimes seen a CSV file. This is hard to edit and operations like adding a field are harder to do.


----------------------------


![slide of information seen before](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-51-and-53.png)


So that's our project.

----------------------------


![slide of points, listed below](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-52.png)

But other ways of running community data projects are available, and we've given you one big reason why you might not want to use Git.

However you choose to run your project, we think it's important to think about the data aspects of it. 

* Will you ask people to send you data?
* Can you reuse existing data?
* How can you keep the data up to date?
* How can you encourage others to use your data?
* How can you encourage others to REuse your data?
* Which existing data standards should you use to publish your data?
* How can you spread the load of editing?
* How can you test and demonstrate the quality of your data?
* Are other platforms already suitable (eg Wikidata, Open Street Map)?
* How can you tie this all together?


[We wrote this blog post that goes into that more](http://web.archive.org/web/20240621164805/https://opendataservices.coop/blog/2020/05/07/community-data.html).

----------------------------


![slide of information seen before](/images/2024-09-datatig-python-talk/2024-09-datatig-python-talk-slide-51-and-53.png)


Thank you!

[Find out more](https://www.datatig.com)


id: 2024-08-11-hub-released
git_filename: _posts/2024-08-11-hub-released.md
format: md
field_title: Hub Released
field_layout: blog
data:
{"layout": "blog", "title": "Hub Released", "content": "This summer we have started a Hub website - a web app that can host the data from many repositories with features to:\n\n* Build the data automatically on any change (checks regularly and responds to webhooks).\n* Allow people to use the data, with browse, search, exports and API.\n* Provide forms for people to edit and create new data, with a markdown editor if applicable.\n* Log in with GitHub and have their changes appear as commits on GitHub automatically.\n* Report on the data - it's correctness and errors, stats about the data and link checking any URLs in the data.\n\nIt supports GitHub, but it can also work with any Git host that provides a public Git point to clone from. \nThe code is structured so it's possible to add support for other hosts like GitLab later.\n\nThis is [now available online at hub.datatig.com](https://hub.datatig.com/).\n\nThe [source code is open source](https://github.com/DataTig/Hub) so you can host this yourself. \nFor [full hosting requirements see the docs](https://datatig-hub.readthedocs.io/en/latest/).\n\nIt's written in Django and uses [our Open Source python library](https://pypi.org/project/DataTig/) heavily. \nWhile developing the Hub we have also developed the library this summer, releasing version 0.6 with new features. \nThe Hub also provides a space to experiment with new features before they are ready to make it into the library. \n(For example link checking is currently only available in the Hub, but will make it into a library release later.)"}

field_content:
This summer we have started a Hub website - a web app that can host the data from many repositories with features to:

* Build the data automatically on any change (checks regularly and responds to webhooks).
* Allow people to use the data, with browse, search, exports and API.
* Provide forms for people to edit and create new data, with a markdown editor if applicable.
* Log in with GitHub and have their changes appear as commits on GitHub automatically.
* Report on the data - it's correctness and errors, stats about the data and link checking any URLs in the data.

It supports GitHub, but it can also work with any Git host that provides a public Git point to clone from. 
The code is structured so it's possible to add support for other hosts like GitLab later.

This is [now available online at hub.datatig.com](https://hub.datatig.com/).

The [source code is open source](https://github.com/DataTig/Hub) so you can host this yourself. 
For [full hosting requirements see the docs](https://datatig-hub.readthedocs.io/en/latest/).

It's written in Django and uses [our Open Source python library](https://pypi.org/project/DataTig/) heavily. 
While developing the Hub we have also developed the library this summer, releasing version 0.6 with new features. 
The Hub also provides a space to experiment with new features before they are ready to make it into the library. 
(For example link checking is currently only available in the Hub, but will make it into a library release later.)


id: 2021-04-15-python-library-version-0-1-1-released
git_filename: _posts/2021-04-15-python-library-version-0-1-1-released.md
format: md
field_title: Python Library Version 0.1.1 Released
field_layout: blog
data:
{"layout": "blog", "title": "Python Library Version 0.1.1 Released", "content": "We released Version 0.1.1 of our Python Library. This immediately replaces version 0.1.0 that was also released today, as this had a Python packaging error.\n\nFor details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#011---2021-04-15)"}

field_content:
We released Version 0.1.1 of our Python Library. This immediately replaces version 0.1.0 that was also released today, as this had a Python packaging error.

For details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#011---2021-04-15)


id: 2020-05-07-how_to_run_a_useful_community_data_project
git_filename: _posts/2020-05-07-how_to_run_a_useful_community_data_project.md
format: md
field_title: How to run a useful community data project
field_layout: blog
data:
{"layout": "blog", "title": "How to run a useful community data project", "content": "In [a post on Open Data Services blog](http://web.archive.org/web/20240621164805/https://opendataservices.coop/blog/2020/05/07/community-data.html)\nwe wrote up a list of things to think about when running a community data project."}

field_content:
In [a post on Open Data Services blog](http://web.archive.org/web/20240621164805/https://opendataservices.coop/blog/2020/05/07/community-data.html)
we wrote up a list of things to think about when running a community data project.


id: 2018-02-12-prototype-launched
git_filename: _posts/2018-02-12-prototype-launched.md
format: md
field_title: Prototype Launched
field_layout: blog
data:
{"layout": "blog", "title": "Prototype Launched", "content": "We launched a prototype of this project and posted about it at the \n[Open Knowledge Foundation forum](https://discuss.okfn.org/t/working-with-community-data-on-github-new-project-datatig/6394)."}

field_content:
We launched a prototype of this project and posted about it at the 
[Open Knowledge Foundation forum](https://discuss.okfn.org/t/working-with-community-data-on-github-new-project-datatig/6394).


id: 2022-02-26-python-library-version-0-3-0-released
git_filename: _posts/2022-02-26-python-library-version-0-3-0-released.md
format: md
field_title: Python Library Version 0.3.0 Released
field_layout: blog
data:
{"layout": "blog", "title": "Python Library Version 0.3.0 Released", "content": "We released Version 0.3.0 of our Python Library.\n\nFor details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#030---2022-02-26)"}

field_content:
We released Version 0.3.0 of our Python Library.

For details of what's new, see [the changelog](https://github.com/DataTig/DataTig/blob/main/CHANGELOG.md#030---2022-02-26)


