The aim of this section is to point you in the right direction to get started within the FedEarthData Service as an end user.
These pages are targeted to users already using the FedEarthData Service. If you want to join, request access via the EOSC Marketplace.
Cloud, Data and HTC/HPC services in C-SCALE currently require separate accounts. Please follow instructions below depending on the service that you are going to access.
The user account to access Cloud and Data services is provided by EGI Check-In. Steps to create your Check-In account can be found in the EGI Documentation.
The user account to access HTC/HPC services is provided by SURF Research Access Manager (or SRAM). Log into SRAM with your preferred Identity Provider and upload your public SSH key to your profile page.
After creating your user account the next step is to join the relevant user group. So far we have created a user group per Use Case. Therefore, search for the given name of your Use Case and join the appropriate group.
The information about user groups for Cloud and Data services is available in EGI Operations Portal. Search for "C-SCALE" to find all the available groups and see some examples below:
If you are not certain which group to join, request access to the pilot group:
User groups in the Cloud federation are also known as Virtual organisations (or VOs), and the links above are called VO ID Cards.
Users will have to visit the enrolment URL to request access. Each petition needs to be approved by the corresponding VO manager. The enrolment URL and the VO manager for each user group can be found in each VO Card.
The information about user groups for HTC/HPC services is available in SRAM Collaborative Organisation (or COs)
The available SRAM COs for C-SCALE are the following:
The link above is the Join Request URL that users will have to visit to request access. Each petition needs to be approved by the corresponding CO Admin.
Below is the list of OpenStack Horizon dashboards participating in the Cloud federation in C-SCALE:
When logging into the Horizon dashboard of OpenStack sites select either OpenID Connect or EGI Check-In in the dropdown menu. Then use your EGI Check-In credentials.
Cloud resources can also be deployed with TOSCA templates and the INDIGO PaaS Orchestrator. Check out the status dashboard in case of issues.
If it is your first time using the service please send the following details:
to indigo-paas-support<at>lists.infn.it to configure the service for your Use Case.
Pre-requisites for the cloud site to be configured with PaaS Orchestrator:
os_distro, os_versionindigo-paas tagindigo-paas-support<at>lists.infn.it:Example on how to configure properties and tags on images:
openstack image set --tag indigo-paas <image-id>
openstack image set --property os_distro=ubuntu --property os_version=20.04 <image-id>
Here are the instructions on how to access the HTC/HPC federation in C-SCALE:
sram-[CO short name]-[SRAM username]ssh -YC sram-[CO short name]-[SRAM username]@delena.surfsara.nlssh <username>@vsc5.vsc.ac.at. Where <username> is provided by EODC IT team.To access openEO platform please visit https://portal.terrascope.be, choose sign-in and then select EduGAIN & social logins.
The openEO endpoint you can use is openeo.vito.be, for instance via Python:
import openeo
connection = openeo.connect("https://openeo.vito.be").authenticate_oidc(provider_id="egi")
Access to openEO is also explained in the openEO documentation.
The interactive notebooks service is based in the EGI Notebooks platform. Thanks to its integration with EGI Check-in (same as the C-SCALE Cloud and Data federation), traversing the federation from your notebooks has been rendered completly seamless.
Example notebooks that show off the use of data resources in the federation are available on GitHub, especially in notebook 1-C-SCALE_Data Discovery.ipynb.
Check the relevant section below to access the documentation for each provider in C-SCALE.
Links to documentation on each Cloud provider:
See the INDIGO PaaS Orchestrator guides.
Links to documentation for HTC/HPC providers:
See documentation about openEO in the following links:
Please use the C-SCALE Helpdesk.
There is also a forum for C-SCALE on GitHub.
Users may request to add or remove computational resources allocated to them by opening a ticket on their respective GitHub repository. All repositories are listed as projects in the C-SCALE GitHub Organisation. Please remember that these are private repositories and you need to request access.
If you do not have a GitHub repository or you do not have access contact us via GitHub discussions.
Here are the requirements for Use Cases requesting to run Matlab workflows in C-SCALE:
Please note that C-SCALE does not plan to cover the costs associated to Matlab licenses.
Note: This section deals with accessing actual data assets once you know their locations. For discovery, refer to the EO Metadata Query Service.
Data providers in FedEarthData are free to offer alternative methods of access, but required to offer HTTP access to their data. Likewise, they are free to offer the data openly with no authentication, but in case user authentication is necessary, providers are required to provide federated authentication with EGI Check-in. Obtaining an account there is explained above.
The authentication mechanism is based on the use of OIDC access tokens. If yu are using a FedEarthData feature, such as the IaaS cloud management interface, the interactive notebooks, or similar, your tokens are probably already present in your system. As a last resort, you can always obtain a valid token manually from the Check-in Token Portal.
Possessing a locator (URL) of an EO asset, you can download it directly with any HTTP client. To authenticate, you must simply include your valid token into the Authorization header of your request. For example, with curl (note label <copy token here> that is to be replaced with the actual token -- left out from here for brevity):
curl -JO -H "Authorization: Bearer <copy token here>" "https://dhr3.cesnet.cz/odata/v1/Products('8594417f-c085-46aa-ab47-88984438f88a')//Nodes(%27S2B_MSIL1C_20230608T093549_N0509_R036_T33UYQ_20230608T102808.SAFE%27)/Nodes(%27GRANULE%27)/Nodes(%27L1C_T33UYQ_A032666_20230608T093840%27)/Nodes(%27IMG_DATA%27)/Nodes(%27T33UYQ_20230608T093549_TCI.jp2%27)/%24value"
Or with a Python requests.get() method (with your token stored in variable access_token):
requests.get("https://dhr3.cesnet.cz/odata/v1/Products('8594417f-c085-46aa-ab47-88984438f88a')//Nodes(%27S2B_MSIL1C_20230608T093549_N0509_R036_T33UYQ_20230608T102808.SAFE%27)/Nodes(%27GRANULE%27)/Nodes(%27L1C_T33UYQ_A032666_20230608T093840%27)/Nodes(%27IMG_DATA%27)/Nodes(%27T33UYQ_20230608T093549_TCI.jp2%27)/%24value", headers={"authorization": "Bearer {}".format(access_token)})
Certain intermediate results of an analytic workflow have the properties of an Earth observation product (item) in its own right. They may be, e.g., pre-processed images with applied filters, index calculation algorithms, or similar. In borderline cases, even pre-downloaded (staged) files that have not been otherwise modified can be considered interim results.
C-SCALE encourages the use of STAC to manage and organize data. Therefore, any time a large cache of EO products is being established, creation of a STAC collection alongside is is strongly encouraged.
The STACTools Project offers tools to generate metadata for numerous types of Earth observation proucts, including all relevant Sentinel missions. If your product is not directly covered by STACTools, you may still use the generated STAC metadata (taking the form of a JSON file) as a template to modify to match your product.
Although it might be concievable to maintain a catalogue as a file on the local filesystem, it is much more appropriate to run a server implementing the STAC API (and potentially other functions on top). C-SCALE made good use, for instance, of the Resto Catalogue. Users are welcome to run their own, but to lower the adoption threshold C-SCALE also offers a managed, user-level instance of the Catalogue.
The Catalogue is intended precisely for the purpose pursued in this Section, i.e., cataloguing interim data items for later use by their producers. It is not, on the other hand, intended for permanent dissemination of your results -- go through C-SCALE to speak to the representatives of datda providers about that.
The user-level catalogue is not currently integrated with the identity federation. If you want to write to it, you need to request access. Follow instructions given herein to request assistance from C-SCALE.
Once you have write access to the catalogue, you may start registering -- and accessing -- your metadata. Resto comes installed with a Resto API Documentation of its own, which gives basic instruction to create, modify, access and download objects from the catalogue.
There is currently no bespoke service to shift large volumes of data across the federation. Typically you need to orchestrate a simple workflow yourself to perform the required task:
For dissemination of your final results, especially if they are of spatio-temporal nature, contact C-SCALE data providers. You will have to negotiate:
Storage location and upload method.
Metadata generation responsibilities.
There is no set procedure -- the resulting agreement will depend on the type and size of your data, as well as the technology employed by the given provider. You may expect to be given a location and credentials/permissions to upload your results, and either to be left with the responsibility to generate metadata for your products and submit them to the providers, or with a promise by the provider to handle metadata downstream by themselves.
Here are the links to the tutorials that C-SCALE has prepared:
Do you have suggestions for new tutorials? Please let us know!
If you run Docker containers on top of Virtual Machines you may encounter unexpected network connectivity issues. If that is the case, please try the steps below:
# check current MTU value
sudo docker network inspect bridge | grep mtu | awk '{print $2}'
# the default 1500 value does not work properly
# edit docker configuration
sudo vi /etc/docker/daemon.json
# ensure this is added
{
"mtu": 1376
}
# then restart docker
sudo systemctl restart docker
We experienced this issue when trying to install a pip dependency using continuumio/miniconda3 container from docker hub.