Core-Admin Administration manual

What’s Core-Admin

Core-Admin is a software platform with a centralized design that allows to control hundred of servers in an easy way. It provides into a single solution support for server administration, configuration, app deployment, internal status checking, file system monitoring, real-time log watching and many more features.

Core-Admin features

Some of the features that Core-Admin provided are:

*A centralized design focused on highly interactive services. You can configure your servers and let your users to also invoke end user applications, limitting their action to certain objects.

*Built-in detailed internal server monitoring system, with self-detection and self-reparation support.

*Secure design that doesn’t require an open port at the destination machine to reach the agent. Thanks to BEEP protocol (http://www.beepcore.org) and Vortex Library implementation (http://www.aspl.es/vortex) agents create a BEEP session which is then used by central server to create independent channels, each of them with its separated security context.

*Lots of ready to use applications focused on covering common tasks in servers.  To name some of them:  global task manager, real time log watching and file-system change monitor, integrated IP blocking system, system users management, process and service viewer, general cron manager..

*Lots of ready to use checkers that will let you check many internal details of your servers. See a full list here.

*A watcher  daemon (crad-log-watcher) that provides real time analysis to find expressions associated to important events (like, for example, disk sector failures which are a sign of hardware failure which aren’t detected by SMART in many cases). This tool also provides content expression inspection which allows, for example, to detect usual hackings in Web pages.

*A powerful and flexible event notification system that will let you generate useful events from your scripts and applications, conveying them along with other events to your core-admin web panel.

Why Core-Admin was created

Core-Admin was written by ASPL (Advanced Software Production Line, S.L. http://www.aspl.es), a spanish company that provides Linux services to install and maintain these systems. At some point, we started to handle hundred of servers and we felt that some problems like disk failures or application level service degration weren’t detected enough fast so we can react by providing a fast reparation solution.

Most of these problems, which start as a simple log error, keep on degrading during several hours until the service is fully broken, reaching finally to the end user. At the same time, we felt that there were many available solutions to provide easy administration services, but all of them were designed to provide a single panel to manage a single machine.

Of course there were some solutions to provide a kind of overlord panel but its design makes administrators using them to embrase bad practises. We needed a solution that provide fast access to all of our servers with a single strong credential but without having to know each server’s administrator credential.

At the same time, we needed a solution to write applications to handle common services in an easy and fast way, which were able to provide an easy to use interface for us and for our end users without having to take care about server’s location.

Core-Admin platform architecture brief

Core-Admin is server/device administration framework that is composed by a client interface (mostly the web-client), a central server and a software agent that runs on each machine the platform controls.

The mision of the Core-Admin’s agent is to provide a persistent conection to the server (so it is not required to open a port on the agent’s machine) and also to work as an invocation service bridge for all commands validated by the central server and requested by core-admin clients.

The mision of the Core-Admin central server is to provide support for users, permission enforcement, machine location and many other services that allows Core-Admin clients to locate and invoke services on agents connected.

Taking all these pieces together (client, server and agent), the platform provides an unified access to all server’s services, limiting access to them to the permissions associated to each user.

Now, on top of this foundation Core-Admin provides applications, checkers and collectors which are in charge of providing the final end-user functionality and at the same time to serve as the basis to extend the platform itself.

Software components and technologies used by Core-Admin

Here are a list of the key software components used by Core-Admin:

* Vortex Library which implement all BEEP communication between agents and central server.

* Turbulence which implement core-admin central server (using mod-python and PyVortex)

* noPoll which provides WebSocket support to Core-Admin central server.

* jsVortex which provides BEEP and BEEP over WebSocket support for the web-client.

* Axl Library which provides XML support to agents and central server (through PyAxl).

* Python as general language used across the platform (at the agent and the central server).

* Dojo toolkit which is used for the visual part of the web-client (Graphical Widgets, Frames, Buttons, etc).

* GNU Sasl which provides support to implement authentication support through SASL, used by Vortex Library and Turbulence, as defined by BEEP.

* OpenSSL which provides TLS support for all connection between central server and agents, as well other cyphering features.

Core-Admin applications, collectors and checkers

On top of the Core-Admin platform run applications, collectors and checkers. They provide the end user functionality and it is the way the Core-Admin platform is extended.

In simple terms, this design allows to free applications, collectors and checkers from dealing with details like users, permissions, object (un)marshalling, etc, letting the central core-admin server to handle these details.

This is a brief description for each of these components:

• Applications: these are the end user apps that are targeted to provide especific features in different areas like Webhosting administration, samba admin or firewall management. These apps may be designed to be run by administrators only (like firewall management) or by end users (like DNS domain managent).
• Collectors: are small apps targeted to gather especific domain data to be shown in an especial section of the machine view. These collectors allows to get, for example, current CPU or disk usage. Generally speaking, collectors are only used by machine or platform administrators.
• Checkers: are small apps used to check especific aspects of machines connected, so they report failures found to the central server and, in some cases, they are able to fix them (by restarting the service, for example). Checker’s engine is run by core-admin’s agent on each machine in an independent manner.

Installing Core-Admin server

Important note before continue. Please, check that your server have the final IP currently configured. This value will be used to configure your Core-Admin setup and it is also used at the license validation phase. Of course you can change this later but that will require from you additional work.

To know your current server IP (the default used for outgoing traffic) run the following command:

Install Core-Admin server (resumed instructions, batch mode)

The following instructions will allow you to install Core-Admin very quickly for your testing and possible in many cases, it will be just enough. For more control over the installation see the “step by step instructions below”.

1. First, install a mysql server if you don’t have it and a set of packages:

   >> apt-get install mysql-server python wget python-mysqldb
   # or the equivalent command for your box and mysql backend
   

2. Now download core-admin-installer:

   >> cd /root
   >> rm -f core-admin-installer.py
   >> wget http://www.core-admin.com/downloads/core-admin-installer.py 
   >> chmod +x core-admin-installer.py
   

3. Install server and agent in one step. Define your so your core-admin panel will be loaded from there, let’s say, core-admin.yourdomain.com:

   >> ./core-admin-installer.py  --server-install --hostname=<HOSTNAME> --license-accepted --assume-yes  --no-license-code
   

4. Now install let’s encrypt tools, request and install let’s encrypt certificate for your server:

   >> ./core-admin-installer.py --core-admin-le-cert=<your-mail-contact>
   

5. Now, your core-admin installation is fully operational. Install some more applications (Webhosting management, Mail admin, DNS admin…):

   >> crad-update.pyc -i webhosting_management
   >> crad-update.pyc -i mail_admin
   >> crad-update.pyc -i dns_admin
   

6. Now connect to http://core-admin.yourdomain.com to start using your server

Installing Core-Admin server (step by step detailed instructions)

Getting Core-Admin server installed on Debian/Ubuntu/Centos systems

Core-Admin central server installation is designed to integrate into your OS packaging system. Please, before proceed, check if your platform is supported here. Once you have selected the machine that will host the Core-Admin server, follow next steps to install Core-Admin server:

1You need to have MySQL server installed and you have to know the administrator password or at least a MySQL user that is able to create and install databases.

2Now, you need some base packages that are required by the core-admin installer:

3Now, download the Core-Admin installer like this:

4Finally, run the installer and choose “Install core-admin server” and follow instructions. During the installation, most of the parameters asked by the installer are obvious, but pay attention to the Core-Admin server name you provide because all the installation will be bound to that name and the web client will be reachable through that name, among other settings.

We stay at point 1. Install core-admin server and click enter.

Next, we accept license terms. We use the position keys to Accept and click enter.

Install Turbulence library, needed to applications executions,

The installation requires to update sources.list file. This operation does not modify the existing data, only add new lines,

Next, we choose the server name for Core-Admin. Ej. core-admin.your-domain.com. And click enter,

Because of the importance of this step, a confirmation is required, clicking Accept. If you want to change the name, you can provide another name,

In case a problem is detected, the system will inform you,

In case no problems were detected, the MySQL configuration is needed. We provide a password for root admin the database and click enter,

Once the database installation is finished, the platform administrator login ir required. You can use words like admin, or administrator, or an email address. This step corresponds to 3. Add a platforrm admin user user (for server) in the main menu,

Next, a password is required for the admin user,

We enter an email address for the admin user and click enter,

In the next step, the suscription license code is required. You can enter it now or later from the panel. We recomend to fill it now. This step corresponds to 7. Install server license code in the main menu,

We enter the suscription license code and click enter,

If everything goes right, a confirmation message is showed. We continue with the instalation by clicking Accept,

Next step is install core-admin agent. This step corresponds to 2. Install core-admin agent in the main menu. We click Ok to continue,

Once the installation is finished, aditional information is showed about how to access to the panel,

Typing the url showed we have the access to our core-admin panel. We use the login and password of the admin user server,

In the top menu we can find the top menu with all the available functions depending on the user that has logged in. We can also find in this menu the name user, date, machines in OK status and machines in failure. The central area is a machines view and applications that are running. In the left side we can find the applications and machines area; depending on the permissions of the user, the number of machines and applications showed will change. And on the rigth side, we can see the systems events; this area is reserved to platforms and machines administrators.

If you got your Core-Admin server successfully installed, now we recommend you to follow the next sections to do an initial setup. Once that is done, you can continue adding Core-Admin agents to start managing servers.

Initial Core-Admin server configurations

After you have successfully install Core-Admin server, please, check the following elements to adjust the platform to your needs.

*Installing your Core-Admin subscription license codes (not required if done during installation) — Before continue, you have to install all subscription licenses you have to enable all features at the Core-Admin server.
The default installation leaves the server ready to function to but only prepared to serve a few requests in order to allow you installing your licenses codes. To install a license code, log-in into the platform as platform administrator and launch the license manager located at the top

If everything is properly loaded, log off and log-in again to have a fresh login with all license codes loaded. If you have any problem with you subscription licenses, please contact us.

In case you have already install the license code in the server installation, you can verify it in

*Now configure which applications you want deployed on first connect by default — By default, when a Core-Admin agent is joined, an automatic software deployment happens. In the case of checkers, these are detected. In the case of applications, only those flagged as “deploy on first connect” are selected in this phase. To configure which applications you want to deploy on first connect, launch the application manager

and then select each application you want by clicking on them.

For example, let’s click on Cert Manager

then check Deploy on first connect and then click on Edit application to save changes.

If you don’t know where to start, we recommend you to leave this setting as is (you can review it later).

You are almost done with the initial setup. Please, review next section about Core-Admin server certificate. After that, you can join your first Core-Admin agent.

Installing a signed CRT (SSL/TLS certificate) to protect Core-Admin server communications

Core-Admin server needs a digital certificate to enable the SSL/TLS protocol that protects all sessions that are established by Core-Admin agents and by Core-Admin users. In this context, Core-Admin installer prepares and setups an unsigned certificate which is fully functional for all Core-Admin agents and for those users connecting using JavaSocketConnector.

See additional information in this article: connecting to Core-Admin

However, if you pretend to provide hosting services or to provide an easy access to your end users, it is recommend to use WebSocket protocol especially because you don’t have to force users without technical skills to go through installing Java and enabling it on their browsers.

Websocket is automatically detected by the core-admin web-client so users won’t notice it. However, all current WebSocket implementations requires a certificate signed by a trusted party.

We are providing certificate signing services to Core-Admin users. Please, check the following page.

To install a signed SSL/TLS certificate, follow next steps:

1  Add Cert Manager application. We click on the machine, our Core-Admin server

We click on Actions and select Show machine config

We go to Aplications tab and cñoc pm Certificate Manager

Ticl on Is selected and save changes by clicking on Edit application

We clic on Close panel

2  Get CSR. In Aplications tab of the machine, we click on Certificates manager,

We select the machine

And click on Get CSR

We choose the signature algorithm and click on Get CSR

We get the certificate request, it has to be sent to our certificator to return us a signed CRT and the CA.

3  CSR installation. Once the certificater send us the information needed, we click on complete CSR and add the information get. Last step is click on Complete CSR to end the certificate installation.

At this point, you can connect directly with the WebSocket without problems.

Installing Core-Admin agent and join it

Before proceed please, check that the system you are about to join is currently supported.

To install a core-admin agent and join it to the server, follow next steps,

1 In the top menu System and clic on Join machine,

We choose a name for the server, and select the option Off-line login, then we clic on Join machine,

The application will show you a string,

2 Copy that string in file located at the new machive, for example,

and from the terminal we execute (as root):

$ core-admin-agent.pyc --join /root/join.txt
$ /etc/init.d/crad-agent restart

Now, your Core-Admin agent is connected to your central server and you also have all the elements needed to your first connection.

Please, be sure Core-Admin server port 602/TCP is not blocked by any firewall and that it is reachable from your agent’s server.

Core-Admin user system

Introduction to Core-Admin permission system

Core admin permission system is designed in a way it allows delegation and allow a particular user to only access to a restricted set of objects or to the entire application, the entire machine or even the entire platform.

Permissions are enforced and verified by the central server before letting them go into the agent and results from agents are filtered (at the agent) by filters created by the server itself without
requiring from the application developer to pay attention to the permission system.

Permission levels: how permissions are structured

Permissions are structure in 4 levels, where the the top level has all permissions and lower level fewer. Here is the levels:

Core-Admin permission level: parent relation

Along with the permission system, it is implicit the parent relation. This means that having machine admin permission in a particular machine, provides access to all conceptual children objects: applications inside this machine and its objects.

The same way, being object owner provides unlimited access to that object and child objects (as defined by the application).

Enforcing plaform and machine admin users

In the case it is required to control and limit platform admin and machine admin users, even though they are created with the right permission, then you have to use the following couple of files that are located at the Core-Admin central server you want to enforce those users:

Each file contains a list of users, one in each line. Now each valid user will be checked and those users appearing on those files will be allowed to log-in as machine or platform admin (according to the file the user was added). In the case a platform admin or machine admin user logs in into the platform and it is not declared inside those files, the log-in operation will fail and a security report will be generated.

Taking further this support, it is possible to completely disable platform admin or machine admin users at all by just creating those files empty. This is a good security feature in the case you want a Core-Admin server without any capability to allow access to machine or platform admins. Then, you can enable access at will by removing those files.

Creating Core-Admin users

To create a new Core-Admin user, no matter if it is a platform admin, machine admin or a hosting user that only has access to certain objects, we first have to create the user. This is done by using the top menu,

Now click on “Users” section to load all current Core-Admin users,

cilck on  ”Add user“,

A user form will appear. Insert login, name, password and mail account and click on Create new core-admin user.

That’s all. Now you have a valid core-admin user. Now, you have to complete previous operation with some additional steps according to the kind of user you need. See next video which shows how to create a new Core-Admin user:

Setting up new Core-Admin platform admin users

For an existing user, click on the user

We click on Platform security (flags), and select Is platform admin. To apply changes, we click on Edit core-admin user.

You only need to create Core-Admin platform admin users when you want a full access granted administrator user. Do not create such user for a normal user because it will be able to access to the whole platform.

IT IS A GOOD practise to test users created (log-in with them) to see what he can see. That way you make really sure that the user create has the rights you want..

To get an explanation about the rest of flags that appears the window, see next.

Core-Admin platform security flags

Core-Admin permission system is composed by two parts. These two parts are the ones to be checked to see what can do the user. These parts are:

* Core-Admin platform security flags (explained next)

* Permissions associated through the permission management (explained later)

Platform security flags allows Core-Admin platform to do certain operations or to allow certain operations to Core-Admin users that aren’t platform admin. Please, click on the help icon  on each permission to get an explanation about it.

This help icon is available at many parts of every application and provides especific help for each parameter. To get that info, just click on it.

Creating machine admin users

Having a user created (see previous section), to assign permission to a certain user to manage a machine or a set of them (and thus become machine admin), click on the top System menu and then click on Manage permissions

It will launch Permission management application which allows to control machine, app and object level permissions asociated to each user. As we so before, permissions in Core-Admin, without considering flags, are handled as a hierarchical manner.

Select the machine where we want to assign permissions to a certain user, in the left tree. his will load all users that are currently machine admin in the selected machine. You can choose from two options:

* Add a new user, by clicking on “Add user“.

We select the user we want to add and click on Accept selection.

* Remove an existing user to remoke its machine admin permission. For that, click on the user and remove it, by clicking on Remove item.

Creating machine application admin and/or object admin users

Please see how are created machine admin users in the previous section. It follows same rules. You have to select the application inside the machine or the object inside the application you want to assign permission, and then add or remove the user as shown in the previous section.

Installing/deploying core-admin applications

Assuming a running Core-Admin server, now you can deploy and install applications into your servers through the agent. That is, to install an application into a server you must have a running Core-Admin there (see previous sections to know how to join a machine).

After that, you have two methods to deploy/install a new application into your servers:

*Using application management application. For this:

1 Start this application located at System top menu -> Manage applications.

2 Then, select the application you want to deploy and then click on “Where is configured” to show current deployment status and what servers/agents have that application installed/deployed

3 Now click on the server you want it have the application, check “Is selected” and then click “Edit machine selected”.The following video shows how to complete these steps:

*Using the machine configuration dialog. For this:

1 Launch the machine admin panel (by click on the machine at the left selection area). If you don’t see your machines click on the header “Machines” to show them. The following video shows how to open the machine admin panel:

2 Now, click on the “Actions” button located at the top right, a rolling menu will appear, then click “Show machine config”.

3 Now go to “Applications” tab and select the application you want to install/deploy to show the dialog, and inside it, click on “Is selected” and then click on “Edit application configured”.Next is shown a video that shows how to complete these steps. At the end of the video, the user clicks on “Deploy application”. That isn’t needed but shows how to force deployment after the was configured.

In both methods the application will be deployed in the server and enable by the central server to be used. In the case the application requires installation, an installation window will appear requesting for initial settings to proceed with the installation (like Mail Admin, Webhosting Management and others..).

Keep also in mind that some applications have no license limits so they can be deployed any number of times, but others requires subscription licenses. In any case, if it is not possible to deploy, an error will indicate the particular error.

Installing/deploying core-admin checkers

By default, checkers are deployed automatically on first agent connection, attempting to auto-detect which checkers can be supported by this system. In the case you want to deploy a certain checker manually or to deploy the checker itself, please, see previous sections about deploy applications. Checkers are deployed the same but under the “Checkers” section.

Configuring checkers

Once a checker is deployed, it is possible to configure it. You can do it in many ways. These are them:

1 When you receive a notification about a certain checker, if the checker admits configuration, you will be able to configure it by clicking on the notification, which will expand available options, and one of them is “Configure checker”.

2 You can configure the checker by openining the checkers monitor, then click to select the machine you want to configure, then select the checker and then click to configure. See the following video to know how:

How to uninstall Core-Admin

You can uinstall core-admin packages and leave its configuration by running the following according to your OS:

# On Debian/Ubuntu systems
$ apt-get remove core-admin-*

# On Centos systems
$ yum remove core-admin-*

If you want to completely remove Core-Admin, including its configurations and databases (for a fresh start):

$ apt-get remove core-admin-*
$ rm -rf /etc/core-admin /var/beep/core-admin /root/core-admin-installer.py

Core-Admin common operations

Core-Admin has a simply and natural user interface to make it usable without having to read a lot of documentation.

Updating Core-admin software

Read next post and see how easy is to get new updates and new applications from your core-admin panel.

Watching logs and filesystem with Core-Admin log watcher (crad-log-watcher)

Blocking IPs with Core-Admin Ip blocker

See next article to know how to block an IP at certain machine with the Core-Admin IP blocker tool.

Disabling notifications at agent level (core-admin-agent)

You can disable all notifications generated by your core-admin agent even for a particular connection.
That way, you can make an agent to disable all notifications done at agent level, without having to
introduce any filter at the core-admin server (or core-admin servers, because an agent can be connected
to several core-admin servers at the same time).

To disable notifications, you have to add skip-notify=’yes’ to master node found at the xml path

/core-admin-client/master-nodes/master

located at the file:

/etc/core-admin/client/client.conf

After that, you have to restart your core-admin agent from the panel or from console:

>> service crad-agent restart

Core-Admin Distribution upgrade

The following are notes about upgrading Core-Admin when a distribution upgrade happens. In general, upgrading core-admin due to a server upgrade involves a) updating Core-Admin packages and b) redeploying applications, checkers and collectors configured to use the new python magic. The following explain how to do that upgrade for different transitions:

Distribution upgrade from lenny to squeeze

1 These are the steps for this distribution upgrade. First, update your /etc/apt/sources.list and replace “lenny” by “squeeze” at the core-admin.com sources. It should look like this:

2 After that, upgrade your distribution list. It shouldn’t give any error associated to www.core-admin.com

3 Now, upgrade base packages with the following commands:

4 Now run the following command to upgrade the rest of the packages:

5 After that, you should have connection to the agent. Now from inside the Core-Admin panel, redeploy applications, collectors and checkers to use new versions associated to the current python magic. For that, select the machine, click on “Actions” (on the top right) and then “Show machine configuration”.

6 Now inside the opened panel, click on Collectors -> redeploy collectors, then click on Applications -> redeploy applications and then click on Checkers -> redeploy checkers.

7 Now, restart Core-Admin log watcher so it can use the new deployed log watcher management application with the folllowing command or clicking on Actions -> Restart Agent (and select to restart log watcher).