Data on projects, partners and calls. What is the process, from collection to post-publication?
This page was last updated on 2025-11-18
Keep.eu contains Interreg project and partner data dating back to the 2000-2006 programming period. With each new period, new regulatory requirements come into effect, new data structures are agreed upon, new technologies emerge, and new workflows must be designed to ensure keep.eu’s data integrity. Interact adapts its data processing procedures to every programming period, covering the entire cycle from collection to post-publication.
This FAQ entry describes the workflow for project and partner data processing in keep.eu for the 2021-2027 programming period only.
Important note: Keep.eu contains only contracted projects.
While the tasks and roles of each team member are described in detail below, any issue that a team member cannot resolve individually must be escalated for discussion within the keep.eu operational team (keep.eu’s data manager, project and product manager, and technical manager).
.
Before it starts: Contacting all Interreg programmes
Keep.eu’s source of data on projects and their partners is always the programmes, which send the data to keep.eu in batches of contracted projects. In the first years of the programming period, Interact instead collected minimum data sets from programme websites, but this is no longer occurring. By default, each programme’s project-partner data are updated every four months.
There are three methods for sending data to keep.eu. Programmes either use Jems, send the data to keep.eu via SFTP in an XML or JSON file generated from their monitoring system (prepared in advance according to the instructions available here), or send the data in an XML, JSON, or Excel file by email.
The keep.eu team contacts all programmes to ensure they are prepared to start providing data to keep.eu, including via Jems, or from other monitoring systems. A description of this process will be added to keep.eu’s FAQ soon. In the latter case, they can set up their monitoring systems to send data in the exact format needed for keep.eu, or they can copy the data to be sent to keep.eu from their monitoring system to Excel.
Important note on distribution of Excel files to programmes: Sending Excel templates to programmes for submitting data to keep.eu is not encouraged, as the data would need to be copied manually by an operator from other systems or files. However, if through their contacts with programmes, the data manager sees no alternative but to send an Excel file to some of them as a means of transferring data to keep.eu, they may use the existing template (here).
.
How it starts: Getting data into keep.eu
Keep.eu has a back office, called Kamut, which can import data in XML, JSON, or Excel. Each column below describes the tasks involved in importing data into keep.eu using each of these methods.
Programmes use Jems
A1. Keep.eu sends a warning to the data manager 14 days before the date of next update (as shown in the programme page in keep.eu), which is typically after four months have passed since the previous update of the programme’s project-partner data, or since the last contact with the programme. On the date of the warning, the data manager generates and downloads a JSON file from the programme’s Jems server. If any issue occurs during file generation, the data manager seeks support from the technical manager, who may contact the programme to resolve the issue and resume the process from this stage.
A2. The data manager imports the JSON file to Kamut. If the file contains no new data, the data manager deletes it. If there are technical difficulties with Kamut, the data manager asks for help from the IT service provider.
A3. The data manager sets the next date of project-partner data importing in keep.eu’s Django, a high-level Python web framework used by keep.eu as a general back-office: Either four months from this date if no new data was imported, or four months and a few weeks from this date if new data was found. This interval is shortened if the contracting of new projects is expected before the four months elapse. If there is no new data at the moment, the process ends here.
A4. The data manager opens a new data-processing Redmine issue, indicating the number of updated projects, new projects, and new partnerships (Redmine is the collaborative-work and project-management platform used by the team working with keep.eu). The JSON file is attached to the issue, together with the link to the Kamut-generated file. The Redmine status at completion of this step is set to New.
A5. The data manager verifies in Kamut if all critical fields are filled out. In case at least 90% of all project records received from the programme at hand have their critical fields filled out, the data manager proceeds with the importing. The critical fields and the remaining mandatory fields are identified as such in keep.eu’s list of 2021–2027 project, partner, and call fields, here. The Redmine status at completion of this step is set to Verified ex ante.
A6. If any critical fields are missing, the data manager requests the missing information from the programme (in case only a few projects miss critical fields, the programme at hand needs to be informed of which ones). The Redmine status at completion of this step is set to Clarification ex ante.
A7. If all mandatory fields are complete, the data manager thanks the programme contacts (detailed in Redmine’s log for that programme) and informs them that the data is being imported and they will be notified once it is published.
A8. The importing and processing of data can now start (either for at least 90% of all project data with all critical fields filled out, for all the project data, or for the data received after step A6). The Redmine status at completion of this step is set to Working.
If there are up to 10% of projects with no data published at this point, the data manager starts a new data import for the projects that were not published due to the lack of critical data, once this data is received from the programme at hand.
Programmes use SFTP
B1. Keep.eu sends a warning to the data manager 14 days before the date of next update (as shown in the programme page in keep.eu), which is typically after four months have passed since the previous update of the programme’s project-partner data, or since the last contact with the programme. If no data-importing Redmine issue is open at this date (see step B3), the data manager emails the programme asking for updated data. If no updates are required, the data manager proceeds to step B4.
B2. Kamut automatically detects a JSON or XML file in the SFTP folder (a cron job is in place to scan all the instances of keep.eu’s FTP folder and transfer to Kamut any incoming files).
Important rule on maximum allowed frequency of data updates: If agreed with Interact beforehand, data may be updated more frequently than every four months. However, the maximum acceptable frequency of data updates is once per month. If a programme sends updates more frequently, the files must be rejected until one month has passed since the last accepted update. In such cases, Interact informs the programme concerned. This rule does not apply to justified ad hoc updates.
B3. Data operators open a new data-importing Redmine issue, indicating the date and time when the file was first listed in the imported files section of Kamut, the numbers of projects to be imported, partners, new projects, and updated projects. They also attach the received JSON or XML file.
B4. The data manager sets the next date of project-partner data importing in Django: Four months and a few weeks from this date. This interval is shortened if the contracting of new projects is expected before the four months elapse. If there is no new data at the moment, the process ends here.
B5. The data manager verifies in Kamut if all critical fields are filled out. In case at least 90% of all project records received from the programme at hand have their critical fields filled out, the data manager proceeds with the importing. The critical fields and the remaining mandatory fields are identified as such in keep.eu’s list of 2021–2027 project, partner, and call fields, here. The Redmine status at completion of this step is set to Verified ex ante.
B6. If any critical fields are missing, the data manager requests the missing information from the programme (in case only a few projects miss critical fields, the programme at hand needs to be informed of which ones). The Redmine status at completion of this step is set to Clarification ex ante.
B7. The data manager verifies the number of new or updated projects and partners, and attaches the Excel file to the Redmine issue at hand. The importing and processing of data can now start (either for at least 90% of all project data with all critical fields filled out, for all the project data, or for the data received after step B5). The Redmine status at completion of this step is set to Working.
If there are up to 10% of projects with no data published at this point, the data manager starts a new data import for the projects that were not published due to the lack of critical data, once this data is received from the programme at hand.
Programmes use email
C1. Keep.eu sends a warning to the data manager 14 days before the date of next update (as shown in the programme page in keep.eu), which is typically after four months have passed since the previous update of the programme’s project-partner data, or since the last contact with the programme. If no new data or data update were received from the programme, the data manager emails the programme asking for updated data. If no updates are required, the data manager proceeds to step C4.
C2. After receiving data from the programme, the data manager opens a new data-processing Redmine issue indicating the number of updated projects, the number of new projects and of new partnerships, and attaches the received file to this new Redmine issue. The Redmine status at completion of this step is set to New.
The rule on maximum allowed frequency of data updates applies.
C3. The data manager imports the file to Kamut and adds to the Redmine the issue link to the Kamut-generated file. If there are technical difficulties with Kamut, the data manager asks for help from the IT service provider.
C4. The data manager sets the next date of project-partner data importing in keep.eu’s Django, a high-level Python web framework used by keep.eu as a general back-office: Either four months from this date if no new data was imported, or four months and a few weeks from this date if new data was found. This interval is shortened if the contracting of new projects is expected before the four months elapse. If there is no new data at the moment, the process ends here.
C5. The data manager verifies in Kamut if all critical fields are filled out. In case at least 90% of all project records received from the programme at hand have their critical fields filled out, the data manager proceeds with the importing. The critical fields and the remaining mandatory fields are identified as such in keep.eu’s list of 2021–2027 project, partner, and call fields, here. The Redmine status at completion of this step is set to Verified ex ante.
C6. If any critical fields are missing, the data manager requests the missing information from the programme (in case only a few projects miss critical fields, the programme at hand needs to be informed of which ones). The Redmine status at completion of this step is set to Clarification ex ante.
C7. If all mandatory fields are complete, the data manager thanks the programme contacts (detailed in Redmine’s log for that programme) and informs them that the data is being imported and they will be notified once it is published.
C8. The importing and processing of data can now start (either for at least 90% of all project data with all critical fields filled out, for all the project data, or for the data received after step A6). The Redmine status at completion of this step is set to Working.
If there are up to 10% of projects with no data published at this point, the data manager starts a new data import for the projects that were not published due to the lack of critical data, once this data is received from the programme at hand.
Importing and processing of data: Guaranteeing data integrity, correct publication and correct meta-data
ABC1. The data operators clean and verify data in Kamut. The Redmine status at completion of this step is set to AdminWorking or ClarificationsNeeded.
Important features in Kamut helping with data cleaning:
a) When the user opens a file, the system automatically checks for the presence of lead partners for each project, and projects with no lead partner will be reported in a warning message. No project can be imported without a lead partner.
b) Geolocation issues and other importing problem messages are shown in file details.
c) Kamut provides a three-column comparison view to compare changes coming with the file to currently editing draft and to data already in production, if any.
d) Project fields included in language detection: Name; description; expected_results; achievements; expected_outputs; delivered_outputs. For less than 8 words Google API is used,and for longer texts langdetect library is used. Fields with less than 5 words are always set as undetermined. Languages can be seen by either opening the item, or by using KAMUT features “languages”, which lists all languages in the file.
Recurring incorrect values that data operators need to correct are as follows:
• Dates that do not follow the YYYY-MM-DD format.
• Figures must be formatted as numbers.
• Values for partner types there are not written in lowercase.
• Organisation types often need to be corrected to fit the keep.eu-accepted (HIT) types.
• Words “”none”” and “”n/a”” as well as all kinds of dashes and commas show without reason and need to be deleted.
• Project partner field is often filled with the word project instead of partner.
• Often programmes fill the organisation status with automatic public instead of just public.
• Often programmes add two or more websites for each project (only one can stay, need to verify which one to keep).
• Part of URLs are often missing.
• Paragraph spacing: Spacings are often inserted in paragraphs.
ABC2. The data manager classifies each project according to one, two or three themes (out of the 42 existing themes), translates any required critical fields into English according to their project life cycle, locks fields legitimately not expected to change, and publishes the data. The Redmine status at completion of this step is set to DataPublished.
Translations for keep.eu are exclusively made at the European Commission’s eTranslation machine translation service (https://commission.europa.eu/resources-partners/etranslation_en). In case the data on the following fields is provided only in languages other than English, they must be translated into English and the corresponding English version of the field needs to be filled out:
• Project summary (project description);
• Expected achievements;
• Actual achievements;
• Programme specific objective;
• Type of intervention.
Other important rules to observe at this stage:
• All thematic fields (Programme specific objective in English, type of intervention) should be locked from the first importing.
• Difficulties with data-importing are logged in Redfield ‘Remarks on data’ in Redmine’s log for that programme.
• Checking the languages of the data for any language that might be wrong needs to be done at this stage.
• In case there are discrepancies bewteen programme data in keep.eu and project data on parameters that are common and which need to be aligned, the data manager asks the product manager to verify the programme data (as a new message in the Data issues forum of keep.eu’s Redmine). Once the product manager verifies the data and replies in the same message in Redmine, the data manger verifies the integrity of the data and proceeds.
ABC3. The data manager checks for inactive partners among the data set sent by the programmes.
A common mistake in the partner data from programmes is that inactive partners are not marked as such (if programmes use Jems, they can set the active tag to false). Hence, Kamut singles out partnerships that may be inactive, and the data manager decides if they should be deleted, because these partners are marked as inactive. It can also be the case that partners with a budget of 0 are inactive, but this is not an automatic assumption to be made. It can happen that they are assimilated or associated, even if not marked as such in the respective data sets. In cases where there are partners with a budget of 0, the data manager needs to ask the programme at hand which of these partners should be deleted.
ABC4. The data manager enables the programme’s LoOp (List of Operations), if not enabled before and if the programme at hand uses keep.eu’s LoOp as its own. The data manager verifies the published data. If any mistakes are found, data can either be corrected in Django or reimported. The Redmine statuses at completion of these steps are set to LoOp verified and DataOK.
ABC5. The data manager confirms or corrects, as needed, the new date set in step A3, B4 or C4.
ABC6. The data manager updates the programme’s project-partner meta-data, by updating in Django the tables that are visible at the bottom of every programme page in keep.eu (such as, for example, 2021 – 2027 Interreg VI-A Austria-Czechia’s page, here). The Redmine status at completion of this step is set to MetadataUpdated.
ABC7. The data manager informs the programme concerned about the newly published data and requests validation. The data manager also indicates which fields the programme still needs to send to keep.eu, either critical fields preventing some projects from being processed and published, or mandatory fields that must eventually be published even if projects can go online before those data are received. The Redmine status at completion of this step is set to ProgrammeInformed.
ABC8. The data manager verifies the actual level of data operation service versus the contracted Service Level Agreement, notes the levels, and closes the Redmine issue. The Redmine status at completion of this step is set to Closed.
ABC9. The json file is automatically deleted in Kamut after it is published. The data manager or the data operators delete in Kamut the files that are not published and which they imported themselves.