- 1Core-Admin Administration manual
- 1.1What's Core-Admin
- 1.2Core-Admin features
- 1.3Why Core-Admin was created
- 1.4Core-Admin platform architecture brief
- 1.5Software components and technologies used by Core-Admin
- 1.6Core-Admin applications, collectors and checkers
- 2Installing Core-Admin server
- 2.1Install Core-Admin server (resumed instructions, batch mode)
- 2.2Installing Core-Admin server (step by step detailed instructions)
- 2.3Getting Core-Admin server installed on Debian/Ubuntu/Centos systems
- 2.4Initial Core-Admin server configurations
- 2.5Installing a signed CRT (SSL/TLS certificate) to protect Core-Admin server communications
- 3Installing Core-Admin agent and join it
- 4Core-Admin user system
- 4.1Introduction to Core-Admin permission system
- 4.1.1Permission levels: how permissions are structured
- 4.1.2Core-Admin permission level: parent relation
- 4.2Enforcing plaform and machine admin users
- 4.3Creating Core-Admin users
- 4.3.1Setting up new Core-Admin platform admin users
- 4.3.2Core-Admin platform security flags
- 4.3.3Creating machine admin users
- 4.3.4Creating machine application admin and/or object admin users
- 5Installing/deploying core-admin applications
- 6Installing/deploying core-admin checkers
- 6.1Configuring checkers
- 7How to uninstall Core-Admin
- 8Core-Admin common operations
- 8.1Updating Core-admin software
- 8.2Watching logs and filesystem with Core-Admin log watcher (crad-log-watcher)
- 8.3Blocking IPs with Core-Admin Ip blocker
- 8.4Disabling notifications at agent level (core-admin-agent)
- 9Core-Admin Distribution upgrade
- 9.1Distribution upgrade from lenny to squeeze
Core-Admin Administration manual
Core-Admin is a software platform with a centralized design that allows to control hundred of servers in an easy way. It provides into a single solution support for server administration, configuration, app deployment, internal status checking, file system monitoring, real-time log watching and many more features.
Some of the features that Core-Admin provided are:
*A centralized design focused on highly interactive services. You can configure your servers and let your users to also invoke end user applications, limitting their action to certain objects.
*Built-in detailed internal server monitoring system, with self-detection and self-reparation support.
*Secure design that doesn’t require an open port at the destination machine to reach the agent. Thanks to BEEP protocol (http://www.beepcore.org) and Vortex Library implementation (http://www.aspl.es/vortex) agents create a BEEP session which is then used by central server to create independent channels, each of them with its separated security context.
*Lots of ready to use applications focused on covering common tasks in servers. To name some of them: global task manager, real time log watching and file-system change monitor, integrated IP blocking system, system users management, process and service viewer, general cron manager..
*Lots of ready to use checkers that will let you check many internal details of your servers. See a full list here.
*A watcher daemon (crad-log-watcher) that provides real time analysis to find expressions associated to important events (like, for example, disk sector failures which are a sign of hardware failure which aren’t detected by SMART in many cases). This tool also provides content expression inspection which allows, for example, to detect usual hackings in Web pages.
*A powerful and flexible event notification system that will let you generate useful events from your scripts and applications, conveying them along with other events to your core-admin web panel.
Why Core-Admin was created
Core-Admin was written by ASPL (Advanced Software Production Line, S.L. http://www.aspl.es), a spanish company that provides Linux services to install and maintain these systems. At some point, we started to handle hundred of servers and we felt that some problems like disk failures or application level service degration weren’t detected enough fast so we can react by providing a fast reparation solution.
Most of these problems, which start as a simple log error, keep on degrading during several hours until the service is fully broken, reaching finally to the end user. At the same time, we felt that there were many available solutions to provide easy administration services, but all of them were designed to provide a single panel to manage a single machine.
Of course there were some solutions to provide a kind of overlord panel but its design makes administrators using them to embrase bad practises. We needed a solution that provide fast access to all of our servers with a single strong credential but without having to know each server’s administrator credential.
At the same time, we needed a solution to write applications to handle common services in an easy and fast way, which were able to provide an easy to use interface for us and for our end users without having to take care about server’s location.
Core-Admin platform architecture brief
Core-Admin is server/device administration framework that is composed by a client interface (mostly the web-client), a central server and a software agent that runs on each machine the platform controls.
The mision of the Core-Admin’s agent is to provide a persistent conection to the server (so it is not required to open a port on the agent’s machine) and also to work as an invocation service bridge for all commands validated by the central server and requested by core-admin clients.
The mision of the Core-Admin central server is to provide support for users, permission enforcement, machine location and many other services that allows Core-Admin clients to locate and invoke services on agents connected.
Taking all these pieces together (client, server and agent), the platform provides an unified access to all server’s services, limiting access to them to the permissions associated to each user.
Now, on top of this foundation Core-Admin provides applications, checkers and collectors which are in charge of providing the final end-user functionality and at the same time to serve as the basis to extend the platform itself.
Software components and technologies used by Core-Admin
Here are a list of the key software components used by Core-Admin:
* Vortex Library which implement all BEEP communication between agents and central server.
* Turbulence which implement core-admin central server (using mod-python and PyVortex)
* noPoll which provides WebSocket support to Core-Admin central server.
* jsVortex which provides BEEP and BEEP over WebSocket support for the web-client.
* Axl Library which provides XML support to agents and central server (through PyAxl).
* Python as general language used across the platform (at the agent and the central server).
* Dojo toolkit which is used for the visual part of the web-client (Graphical Widgets, Frames, Buttons, etc).
* GNU Sasl which provides support to implement authentication support through SASL, used by Vortex Library and Turbulence, as defined by BEEP.
* OpenSSL which provides TLS support for all connection between central server and agents, as well other cyphering features.
Core-Admin applications, collectors and checkers
On top of the Core-Admin platform run applications, collectors and checkers. They provide the end user functionality and it is the way the Core-Admin platform is extended.
In simple terms, this design allows to free applications, collectors and checkers from dealing with details like users, permissions, object (un)marshalling, etc, letting the central core-admin server to handle these details.
This is a brief description for each of these components:
- Applications: these are the end user apps that are targeted to provide especific features in different areas like Webhosting administration, samba admin or firewall management. These apps may be designed to be run by administrators only (like firewall management) or by end users (like DNS domain managent).
- Collectors: are small apps targeted to gather especific domain data to be shown in an especial section of the machine view. These collectors allows to get, for example, current CPU or disk usage. Generally speaking, collectors are only used by machine or platform administrators.
- Checkers: are small apps used to check especific aspects of machines connected, so they report failures found to the central server and, in some cases, they are able to fix them (by restarting the service, for example). Checker’s engine is run by core-admin’s agent on each machine in an independent manner.
Installing Core-Admin server
To know your current server IP (the default used for outgoing traffic) run the following command:
> ip route get 220.127.116.11 | head -n 1
18.104.22.168 via 192.168.1.100 dev eth0 src __your_source_ip_will_be_here__
Install Core-Admin server (resumed instructions, batch mode)
The following instructions will allow you to install Core-Admin very quickly for your testing and possible in many cases, it will be just enough. For more control over the installation see the “step by step instructions below”.
- First, install a mysql server if you don’t have it and a set of packages:
>> apt-get install mysql-server python wget python-mysqldb # or the equivalent command for your box and mysql backend
- Now download core-admin-installer:
>> cd /root >> rm -f core-admin-installer.py >> wget http://www.core-admin.com/downloads/core-admin-installer.py >> chmod +x core-admin-installer.py
- Install server and agent in one step. Define your
so your core-admin panel will be loaded from there, let’s say, core-admin.yourdomain.com:
>> ./core-admin-installer.py --server-install --hostname=<HOSTNAME> --license-accepted --assume-yes --no-license-code
- Now install let’s encrypt tools, request and install let’s encrypt certificate for your server:
>> ./core-admin-installer.py --core-admin-le-cert=<your-mail-contact>
- Now, your core-admin installation is fully operational. Install some more applications (Webhosting management, Mail admin, DNS admin…):
>> crad-update.pyc -i webhosting_management >> crad-update.pyc -i mail_admin >> crad-update.pyc -i dns_admin
- Now connect to http://core-admin.yourdomain.com to start using your server
Installing Core-Admin server (step by step detailed instructions)
Getting Core-Admin server installed on Debian/Ubuntu/Centos systems
Core-Admin central server installation is designed to integrate into your OS packaging system. Please, before proceed, check if your platform is supported here. Once you have selected the machine that will host the Core-Admin server, follow next steps to install Core-Admin server:
1You need to have MySQL server installed and you have to know the administrator password or at least a MySQL user that is able to create and install databases.
>> apt-get install mysql-server
2Now, you need some base packages that are required by the core-admin installer:
> apt-get install python wget python-mysqldb
3Now, download the Core-Admin installer like this:
> cd /root
> rm -f core-admin-installer.py
> wget http://www.core-admin.com/downloads/core-admin-installer.py
> chmod +x core-admin-installer.py
4Finally, run the installer and choose “Install core-admin server” and follow instructions. During the installation, most of the parameters asked by the installer are obvious, but pay attention to the Core-Admin server name you provide because all the installation will be bound to that name and the web client will be reachable through that name, among other settings.
We stay at point 1. Install core-admin server and click enter.
Next, we accept license terms. We use the position keys to Accept and click enter.
Install Turbulence library, needed to applications executions,
The installation requires to update sources.list file. This operation does not modify the existing data, only add new lines,
Next, we choose the server name for Core-Admin. Ej. core-admin.your-domain.com. And click enter,
Because of the importance of this step, a confirmation is required, clicking Accept. If you want to change the name, you can provide another name,
In case a problem is detected, the system will inform you,
In case no problems were detected, the MySQL configuration is needed. We provide a password for root admin the database and click enter,
Once the database installation is finished, the platform administrator login ir required. You can use words like admin, or administrator, or an email address. This step corresponds to 3. Add a platforrm admin user user (for server) in the main menu,
Next, a password is required for the admin user,
We enter an email address for the admin user and click enter,
In the next step, the suscription license code is required. You can enter it now or later from the panel. We recomend to fill it now. This step corresponds to 7. Install server license code in the main menu,
We enter the suscription license code and click enter,
If everything goes right, a confirmation message is showed. We continue with the instalation by clicking Accept,
Next step is install core-admin agent. This step corresponds to 2. Install core-admin agent in the main menu. We click Ok to continue,
Once the installation is finished, aditional information is showed about how to access to the panel,
Typing the url showed we have the access to our core-admin panel. We use the login and password of the admin user server,
In the top menu we can find the top menu with all the available functions depending on the user that has logged in. We can also find in this menu the name user, date, machines in OK status and machines in failure. The central area is a machines view and applications that are running. In the left side we can find the applications and machines area; depending on the permissions of the user, the number of machines and applications showed will change. And on the rigth side, we can see the systems events; this area is reserved to platforms and machines administrators.
Initial Core-Admin server configurations
After you have successfully install Core-Admin server, please, check the following elements to adjust the platform to your needs.
*Installing your Core-Admin subscription license codes (not required if done during installation) — Before continue, you have to install all subscription licenses you have to enable all features at the Core-Admin server.
The default installation leaves the server ready to function to but only prepared to serve a few requests in order to allow you installing your licenses codes. To install a license code, log-in into the platform as platform administrator and launch the license manager located at the top
System Menu → License Manager → Options → Add license code
If everything is properly loaded, log off and log-in again to have a fresh login with all license codes loaded. If you have any problem with you subscription licenses, please contact us.
In case you have already install the license code in the server installation, you can verify it in
System Menu → License Manager → License codes
*Now configure which applications you want deployed on first connect by default — By default, when a Core-Admin agent is joined, an automatic software deployment happens. In the case of checkers, these are detected. In the case of applications, only those flagged as “deploy on first connect” are selected in this phase. To configure which applications you want to deploy on first connect, launch the application manager
System menu → Manage applications
and then select each application you want by clicking on them.
For example, let’s click on Cert Manager
then check Deploy on first connect and then click on Edit application to save changes.
If you don’t know where to start, we recommend you to leave this setting as is (you can review it later).
Installing a signed CRT (SSL/TLS certificate) to protect Core-Admin server communications
Core-Admin server needs a digital certificate to enable the SSL/TLS protocol that protects all sessions that are established by Core-Admin agents and by Core-Admin users. In this context, Core-Admin installer prepares and setups an unsigned certificate which is fully functional for all Core-Admin agents and for those users connecting using JavaSocketConnector.
However, if you pretend to provide hosting services or to provide an easy access to your end users, it is recommend to use WebSocket protocol especially because you don’t have to force users without technical skills to go through installing Java and enabling it on their browsers.
Websocket is automatically detected by the core-admin web-client so users won’t notice it. However, all current WebSocket implementations requires a certificate signed by a trusted party.
To install a signed SSL/TLS certificate, follow next steps:
1 Add Cert Manager application. We click on the machine, our Core-Admin server
We click on Actions and select Show machine config
We go to Aplications tab and cñoc pm Certificate Manager
Ticl on Is selected and save changes by clicking on Edit application
We clic on Close panel
2 Get CSR. In Aplications tab of the machine, we click on Certificates manager,
We select the machine
And click on Get CSR
We choose the signature algorithm and click on Get CSR
We get the certificate request, it has to be sent to our certificator to return us a signed CRT and the CA.
3 CSR installation. Once the certificater send us the information needed, we click on complete CSR and add the information get. Last step is click on Complete CSR to end the certificate installation.
Installing Core-Admin agent and join it
To install a core-admin agent and join it to the server, follow next steps,
1 In the top menu System and clic on Join machine,
We choose a name for the server, and select the option Off-line login, then we clic on Join machine,
The application will show you a string,
2 Copy that string in file located at the new machive, for example,
and from the terminal we execute (as root):
$ core-admin-agent.pyc --join /root/join.txt $ /etc/init.d/crad-agent restart
Core-Admin user system
Introduction to Core-Admin permission system
Core admin permission system is designed in a way it allows delegation and allow a particular user to only access to a restricted set of objects or to the entire application, the entire machine or even the entire platform.
Permissions are enforced and verified by the central server before letting them go into the agent and results from agents are filtered (at the agent) by filters created by the server itself without
requiring from the application developer to pay attention to the permission system.
Permission levels: how permissions are structured
Permissions are structure in 4 levels, where the the top level has all permissions and lower level fewer. Here is the levels:
*Platform admin: this is a flag set to a user account that provides all permissions (no restriction). The platform admin is the only one that can provide platform admin rights to other users. This permission is similar to the root unix user or Administrator under windows platform. A part from that, there are some operations that are reserved only for platform admins like join/unjoin machines.
*Machine admin: this is a permission that a platform admin or a user with “can assign machine admin” permission can grant to other user but limited to the machines he owns (at the user’s settings, inside Users management application). This permission allows to do all invocations on the particular machine without restriction.
*Application admin: this is a permission that a platform admin, machine admin or application admin with “can assign application admin” permission can grant to other users but limited to applications he owns (at the user’s settings, inside Users management application).This permission allows to do all invocations on a particular application inside a particular host without restrition.
*Object admin: this is a permission that a platform admin, machine admin or application admin with “can assign object admin” permission can grant to other users and limited to those objects he owns. This permission allows to do all invocations on a particuar application inside a particular host, involving a particular object without restriction (and its children objects, defined by the application itself, that is, is the application that reports that relations and that are then enforced by the Core-Admin server).
Core-Admin permission level: parent relation
Along with the permission system, it is implicit the parent relation. This means that having machine admin permission in a particular machine, provides access to all conceptual children objects: applications inside this machine and its objects.
The same way, being object owner provides unlimited access to that object and child objects (as defined by the application).
Enforcing plaform and machine admin users
In the case it is required to control and limit platform admin and machine admin users, even though they are created with the right permission, then you have to use the following couple of files that are located at the Core-Admin central server you want to enforce those users:
Each file contains a list of users, one in each line. Now each valid user will be checked and those users appearing on those files will be allowed to log-in as machine or platform admin (according to the file the user was added). In the case a platform admin or machine admin user logs in into the platform and it is not declared inside those files, the log-in operation will fail and a security report will be generated.
Taking further this support, it is possible to completely disable platform admin or machine admin users at all by just creating those files empty. This is a good security feature in the case you want a Core-Admin server without any capability to allow access to machine or platform admins. Then, you can enable access at will by removing those files.
Creating Core-Admin users
To create a new Core-Admin user, no matter if it is a platform admin, machine admin or a hosting user that only has access to certain objects, we first have to create the user. This is done by using the top menu,
System > Manage users
Now click on “Users” section to load all current Core-Admin users,
cilck on ”Add user“,
A user form will appear. Insert login, name, password and mail account and click on Create new core-admin user.
Setting up new Core-Admin platform admin users
For an existing user, click on the user
We click on Platform security (flags), and select Is platform admin. To apply changes, we click on Edit core-admin user.
To get an explanation about the rest of flags that appears the window, see next.
Core-Admin platform security flags
Core-Admin permission system is composed by two parts. These two parts are the ones to be checked to see what can do the user. These parts are:
* Core-Admin platform security flags (explained next)
* Permissions associated through the permission management (explained later)
Platform security flags allows Core-Admin platform to do certain operations or to allow certain operations to Core-Admin users that aren’t platform admin. Please, click on the help icon on each permission to get an explanation about it.
This help icon is available at many parts of every application and provides especific help for each parameter. To get that info, just click on it.
Creating machine admin users
Having a user created (see previous section), to assign permission to a certain user to manage a machine or a set of them (and thus become machine admin), click on the top System menu and then click on Manage permissions
System → Manage permissions
It will launch Permission management application which allows to control machine, app and object level permissions asociated to each user. As we so before, permissions in Core-Admin, without considering flags, are handled as a hierarchical manner.
Select the machine where we want to assign permissions to a certain user, in the left tree. his will load all users that are currently machine admin in the selected machine. You can choose from two options:
* Add a new user, by clicking on “Add user“.
We select the user we want to add and click on Accept selection.
* Remove an existing user to remoke its machine admin permission. For that, click on the user and remove it, by clicking on Remove item.
Creating machine application admin and/or object admin users
Please see how are created machine admin users in the previous section. It follows same rules. You have to select the application inside the machine or the object inside the application you want to assign permission, and then add or remove the user as shown in the previous section.
Installing/deploying core-admin applications
Assuming a running Core-Admin server, now you can deploy and install applications into your servers through the agent. That is, to install an application into a server you must have a running Core-Admin there (see previous sections to know how to join a machine).
After that, you have two methods to deploy/install a new application into your servers:
*Using application management application. For this:
1 Start this application located at System top menu -> Manage applications.
2 Then, select the application you want to deploy and then click on “Where is configured” to show current deployment status and what servers/agents have that application installed/deployed
3 Now click on the server you want it have the application, check “Is selected” and then click “Edit machine selected”.The following video shows how to complete these steps:
*Using the machine configuration dialog. For this:
1 Launch the machine admin panel (by click on the machine at the left selection area). If you don’t see your machines click on the header “Machines” to show them. The following video shows how to open the machine admin panel:
2 Now, click on the “Actions” button located at the top right, a rolling menu will appear, then click “Show machine config”.
3 Now go to “Applications” tab and select the application you want to install/deploy to show the dialog, and inside it, click on “Is selected” and then click on “Edit application configured”.Next is shown a video that shows how to complete these steps. At the end of the video, the user clicks on “Deploy application”. That isn’t needed but shows how to force deployment after the was configured.
In both methods the application will be deployed in the server and enable by the central server to be used. In the case the application requires installation, an installation window will appear requesting for initial settings to proceed with the installation (like Mail Admin, Webhosting Management and others..).
Keep also in mind that some applications have no license limits so they can be deployed any number of times, but others requires subscription licenses. In any case, if it is not possible to deploy, an error will indicate the particular error.
Installing/deploying core-admin checkers
By default, checkers are deployed automatically on first agent connection, attempting to auto-detect which checkers can be supported by this system. In the case you want to deploy a certain checker manually or to deploy the checker itself, please, see previous sections about deploy applications. Checkers are deployed the same but under the “Checkers” section.
Once a checker is deployed, it is possible to configure it. You can do it in many ways. These are them:
1 When you receive a notification about a certain checker, if the checker admits configuration, you will be able to configure it by clicking on the notification, which will expand available options, and one of them is “Configure checker”.
2 You can configure the checker by openining the checkers monitor, then click to select the machine you want to configure, then select the checker and then click to configure. See the following video to know how:
How to uninstall Core-Admin
You can uinstall core-admin packages and leave its configuration by running the following according to your OS:
# On Debian/Ubuntu systems $ apt-get remove core-admin-* # On Centos systems $ yum remove core-admin-*
If you want to completely remove Core-Admin, including its configurations and databases (for a fresh start):
$ apt-get remove core-admin-* $ rm -rf /etc/core-admin /var/beep/core-admin /root/core-admin-installer.py
Core-Admin common operations
Core-Admin has a simply and natural user interface to make it usable without having to read a lot of documentation.
Updating Core-admin software
Watching logs and filesystem with Core-Admin log watcher (crad-log-watcher)
Blocking IPs with Core-Admin Ip blocker
See next article to know how to block an IP at certain machine with the Core-Admin IP blocker tool.
Disabling notifications at agent level (core-admin-agent)
You can disable all notifications generated by your core-admin agent even for a particular connection.
That way, you can make an agent to disable all notifications done at agent level, without having to
introduce any filter at the core-admin server (or core-admin servers, because an agent can be connected
to several core-admin servers at the same time).
To disable notifications, you have to add skip-notify=’yes’ to master node found at the xml path
located at the file:
After that, you have to restart your core-admin agent from the panel or from console:
>> service crad-agent restart
Core-Admin Distribution upgrade
The following are notes about upgrading Core-Admin when a distribution upgrade happens. In general, upgrading core-admin due to a server upgrade involves a) updating Core-Admin packages and b) redeploying applications, checkers and collectors configured to use the new python magic. The following explain how to do that upgrade for different transitions:
Distribution upgrade from lenny to squeeze
1 These are the steps for this distribution upgrade. First, update your /etc/apt/sources.list and replace “lenny” by “squeeze” at the core-admin.com sources. It should look like this:
> deb http://www.core-admin.com/debian squeeze/
2 After that, upgrade your distribution list. It shouldn’t give any error associated to www.core-admin.com
>> apt-get update
3 Now, upgrade base packages with the following commands:
>> apt-get –reinstall install core-admin-agent core-admin-agent-common core-admin-common python-axl python-vortex python-vortex-tls python-vortex-sasl python-vortex-alive
4 Now run the following command to upgrade the rest of the packages:
>> crad-update.pyc -e
5 After that, you should have connection to the agent. Now from inside the Core-Admin panel, redeploy applications, collectors and checkers to use new versions associated to the current python magic. For that, select the machine, click on “Actions” (on the top right) and then “Show machine configuration”.
6 Now inside the opened panel, click on Collectors -> redeploy collectors, then click on Applications -> redeploy applications and then click on Checkers -> redeploy checkers.
7 Now, restart Core-Admin log watcher so it can use the new deployed log watcher management application with the folllowing command or clicking on Actions -> Restart Agent (and select to restart log watcher).
>> /etc/init.d/crad-log-watcher restart