和往常一樣，OpenBMC中官方對Redfish 的Log 機制已經有很詳細的介紹，所以這篇是結合一些額外的資料來看整個LogService 的架構和原理 (*如果之後OpenBMC架構有變，這邊不會再更新)

docs/redfish-logging-in-bmcweb.md at master · openbmc/docs · GitHub

LogService

在Redfish 還沒問世之前，BMC的世界中會將有問題的事件紀錄成System Event Log (SEL)，SEL 在IPMI中是用16 Byte來表示，IPMItool 等工具會將其轉換為可讀性高的文字，而在Redfish 協議中，系統事件的紀錄(開關機，溫度異常等)是由LogServices來負責監控和配置這些事件，OpenBMC將這些系統事件稱Event Log (LogServices 不只有系統事件，還有包含audit log, crashdump 等，這篇文章只會介紹EventLog)

Journal - Systemd日誌

Systemd has its own logging system called the journal; running a separate logging daemon is not required systemd/Journal - ArchWiki (archlinux.org)

Journal是systemd本身自帶的日誌系統，它的log分類是根據Syslog協定，透過 "journalctl"指令可以讀取系統日誌，透過設定檔 /etc/systemd/journald.conf 可以更改Journal 設定

//透過 "journalctl"指令可以讀取系統日誌
> journalctl
Jun 25 08:41:09 intel-obmc bmcweb[370]: pam_succeed_if(webserver:auth): requirement "user ingroup redfish" was met by user "root"

Rsyslog是一個基於Syslog 的開源軟體程式，可在設定檔 /etc/rsyslog.conf 中，根據Journal不同的fields 來將不同的logs存放在相對應的檔案和設定輸出格式。

//加上-o json可以將所有filed用json格式印出來，可以看到"PRIORITY" 這樣就算一個filed，我們也可以自己定義想要的filed，filed的數量並沒有規定，名字可以自定義的
 
> journalctl  -o json 
{
    "_SOURCE_REALTIME_TIMESTAMP": "1624610486273832",
    "PRIORITY": "6",
    "_BOOT_ID": "7f34e1dd99bd4d82add8c4f627da6d2c",
    "_SYSTEMD_INVOCATION_ID": "3fb0bea4b7ef4285bda15cde5c400b56",
    "SYSLOG_FACILITY": "10",
    "MESSAGE": "pam_succeed_if(webserver:auth): requirement \"user ingroup redfish\" was met by user \"root\"",
    "__MONOTONIC_TIMESTAMP": "105559270358",
    "_CMDLINE": "/usr/bin/bmcweb",
    "SYSLOG_IDENTIFIER": "bmcweb",
    "_CAP_EFFECTIVE": "1ffffffffff",
    "__CURSOR": "s=7ece329c23f3447497b13bfd083c9ab9;i=534;b=7f34e1dd99bd4d82add8c4f627da6d2c;m=1893d2a3d6;t=5c5931aeec754;x=a43f04e856ee1bf",
    "_HOSTNAME": "intel-obmc",
    "_SYSTEMD_SLICE": "system.slice",
    "_COMM": "bmcweb",
    "_GID": "0",
    "_SYSTEMD_UNIT": "bmcweb.service",
    "_UID": "0",
    "SYSLOG_TIMESTAMP": "Jun 25 08:41:26 ",
    "_TRANSPORT": "syslog",
    "_MACHINE_ID": "0c1eaebf149d400b851c027c0a218f06",
    "_EXE": "/usr/bin/bmcweb",
    "__REALTIME_TIMESTAMP": "1624610486273876",
    "_SYSTEMD_CGROUP": "/system.slice/bmcweb.service",
    "_PID": "370"
}

有journal log和filed的概念後，我們來看openbmc中的Rsyslog怎麼來分類log，底下是openbmc 中/etc/rsyslog.conf的部分內容

openbmc/rsyslog.conf at master · openbmc/openbmc · GitHub

# Template for Redfish messages
# "<timestamp> <MessageId>,<MessageArgs>"
template(name="RedfishTemplate" type="list") {
    property(name="timereported" dateFormat="rfc3339")
    constant(value=" ")
    property(name="$!REDFISH_MESSAGE_ID")
    constant(value=",")
    property(name="$!REDFISH_MESSAGE_ARGS")
    constant(value="\n")
}
 
# If the journal entry has a Redfish MessageId, save as a Redfish event
if ($!REDFISH_MESSAGE_ID != "") then {
   action(type="omfile" file="/var/log/redfish" template="RedfishTemplate")
}

分類步驟

1. 判斷log有無REDFISH_MESSAGE_ID這個filed

# If the journal entry has a Redfish MessageId, save as a Redfish event
if ($!REDFISH_MESSAGE_ID != "") then {
   action(type="omfile" file="/var/log/redfish" template="RedfishTemplate")
}

2. 用redfishtemplate格式存入/var/log/redfish中

template(name="RedfishTemplate" type="list") {
    property(name="timereported" dateFormat="rfc3339")
    constant(value=" ")
    property(name="!REDFISHMESSAGEID")constant(value=",")property(name="!REDFISH_MESSAGE_ARGS")
    constant(value="\n")
}

3. redfish會去/var/log/redfish中撈資料（GET： /redfish/v1/Systems/system/LogServices/EventLog/Entries/）

// redfish-core\lib\log_services.hpp
static bool
    getRedfishLogFiles(std::vector<std::filesystem::path>& redfishLogFiles)
{
    static const std::filesystem::path redfishLogDir = "/var/log";
    static const std::string redfishLogFilename = "redfish";
    ...
}
 
inline void requestRoutesJournalEventLogEntryCollection(App& app)
{
    BMCWEB_ROUTE(app,
                 "/redfish/v1/Systems/system/LogServices/EventLog/Entries/")
        .privileges(redfish::privileges::getLogEntryCollection)
        .methods(boost::beast::http::verb::get)(
           ...
                getRedfishLogFiles(redfishLogFiles);
           ...)
}

新增redfish log可以透過sd_journal_send ，例如

(REDFISH_MESSAGE_ID是必要的，REDFISH_MESSAGE_ARGS則依情況而定）

sd_journal_send("MESSAGE=%s", "journal text", "PRIORITY=%i", <LOG_LEVEL>,
                "REDFISH_MESSAGE_ID=%s",
                "ResourceEvent.1.0.ResourceErrorThresholdExceeded",
                "REDFISH_MESSAGE_ARGS=%s,%d", "Property Name",
                propertyValue, NULL);

Message Registry

REDFISH_MESSAGE_ID 和 REDFISH_MESSAGE_ARGS要填的內容就要參考Message registry

我們先來看一個registry的範例 [GET] /redfish/v1/Registries/OpenBMC/OpenBMC

bmcweb/openbmc_message_registry.hpp at master · openbmc/bmcweb · GitHub

{
    "@Redfish.Copyright": "Copyright 2018 OpenBMC. All rights reserved.",
    "@odata.type": "#MessageRegistry.v1_4_0.MessageRegistry",
    "Description": "This registry defines the base messages for OpenBMC.",
    "Id": "OpenBMC.0.1.0",
    "Language": "en",
    "Messages": {
        "SystemPowerOnFailed": {
            "Description": "Indicates that the system failed to power on.",
            "Message": "System Power-On Failed.",
            "MessageSeverity": "Critical",
            "NumberOfArgs": 0,
            "Resolution": "None.",
            "Severity": "Critical"
        }},
        "VoltageRegulatorOverheated": {
            "Description": "Indicates that the specified voltage regulator 
                             overheated.",
            "Message": "%1 Voltage Regulator Overheated.",
            "MessageSeverity": "Critical",
            "NumberOfArgs": 1,
            "ParamTypes": ["string"],
            "Resolution": "None.",
            "Severity": "Critical"
        }
    },
    "Name": "OpenBMC Message Registry",
    "OwningEntity": "OpenBMC",
    "RegistryPrefix": "OpenBMC",
    "RegistryVersion": "0.1.0"
}

Message registry 是Redfish的一個Resource，中文翻譯叫"註冊表"，簡單來說就是Redfish 會將它想傳遞的訊息（例如EventLog）先註冊在Message registry中，在runtime的時候會透過MessageId （REDFISH_MESSAGE_ID）來將相對應的訊息輸出。底下是redfish spec對MessageId的定義

一個MessageId組成對應registry大概如下

如果想要記個 VoltageRegulatorOverheated的event，

REDFISH_MESSAGE_ID = OpenBMC.0.1.VoltageRegulatorOverheated
REDFISH_MESSAGE_ARGS可以帶"CPU 0"

程式碼寫法如下

sd_journal_send("MESSAGE=%s", "journal text", "PRIORITY=%i", <LOG_LEVEL>,
                "REDFISH_MESSAGE_ID=%s",
                "OpenBMC.0.1.VoltageRegulatorOverheated",
                "REDFISH_MESSAGE_ARGS=%s", "CPU 0", NULL);

cat /var/log/redfish

1970-01-01T00:00:15.077204+00:00 OpenBMC.0.1.VoltageRegulatorOverheated,CPU 0

Redfish 會透過REDFISH_MESSAGE_ID 將Event log翻譯成可讀性高的文字訊息，其中log的一些property是可以和剛剛上面提的message prefix相對應的

{
        "@odata.id": "/redfish/v1/Systems/system/LogServices/EventLog/Entries/15_2",
        "@odata.type": "#LogEntry.v1_4_0.LogEntry",
        "Created": "1970-01-01T00:00:15+00:00",
        "EntryType": "Event",
        "Id": "15_2",
        "Message": "CPU 0 Voltage Regulator Overheated.",
        "MessageArgs": [
            "CPU 0"
        ],
        "MessageId": "OpenBMC.0.1.VoltageRegulatorOverheated",
        "Name": "System Event Log Entry",
        "Severity": "Critical"
}

目前openBMC支援的prefix有Base, TaskEvents, ResourceEvent, OpenBMC四種，可以在source code:bmweb\redfish-core\include\registries"或透過GET /redfish/v1/Registries 看到

可以透過 /redfish/v1/Systems/system/LogServices/EventLog/Entries 取得Event Log

EventService

我們先來看EventService的架構

EventService是可以偵測有沒有sel 新增，並即時的通知使用者

目前有兩種訂閱(subscribe)方式

RedfishEvent
SSE

RedfishEvent

訂閱方式是透過[POST]/redfish/v1/EventService/Subscriptions with body:

{

"Context": "CustomText",

"Destination": "https://<LISTENER_IP>:<PORT>/<URI>",

"Protocol": "Redfish",

"EventFormatType": "Event",

"DeliveryRetryPolicy": "RetryForever",

"HttpHeaders": [

{

"Authentication": "Splunk 49ce1-sdfoijf-wegsdfe-qwessddasd"

}

]

}

這邊BMC是算client端，它會連上我們指定的server(Destination)，並且在有新的sel出現的時候，push data給指定的server端，那我們對server端的驗證是透過帶入的HttpHeaders。如果剛剛有訂閱成功，我們可以在/redfish/v1/EventService/Subscriptions/{id} 看到我們剛剛設定的值

#/redfish/v1/EventService/Subscriptions/553916031
{
    "@odata.id": "/redfish/v1/EventService/Subscriptions/553916031",
    "@odata.type": "#EventDestination.v1_7_0.EventDestination",
    "Context": "CustomText",
    "DeliveryRetryPolicy": "RetryForever",
    "Destination": "https://10.142.24.24:8000/",
    "EventFormatType": "Event",
    "HttpHeaders": [
        {
            "Authentication": "Splunk 49ce1-sdfoijf-wegsdfe-qwessddasd"
        }
    ],
    "Id": "553916031",
    "MessageIds": [],
    "MetricReportDefinitions": [],
    "Name": "Event Destination 553916031",
    "Protocol": "Redfish",
    "RegistryPrefixes": [],
    "ResourceTypes": [],
    "SubscriptionType": "RedfishEvent"
}

SSE(Server-Sent Events)

它算是一種比較新的網路溝通協定，當server-client建立連線後，server端就可以一直push data給client端，這邊bmc就是做為server端

因此我們只需要瀏覽器中輸入SSE URI (/redfish/v1/EventService/Subscriptions/SSE )，連上BMC之後，就可以不定時收到新增的sel

那EventService也有對Subscriptions提供一些filter

[Get] /redfish/v1/EventService 中的SSEFilterPropertiesSupported

{
    "SSEFilterPropertiesSupported": {
        "EventFormatType": true,
        "MessageId": true,
        "MetricReportDefinition": true,
        "OriginResource": false,
        "RegistryPrefix": true,
        "ResourceType": false
    },
}

可以知道目前bmc提供那些filter選項給我們選，目前有EventFormatType, MessageId, MetricReportDefinition, RegistryPrefix。

因為RegistryPrefix和MessageId兩個是衝突的，所以不能同時設定

那我們要建立SSE 還要順便新增filter的選項，就可以透過odata 的$filter query option

我們來看一下odata的URI Components

ServicesRoot就是/redfish/v1

ResourcePath就是/EventService/Subscriptions/SSE

那要新增QueryOptions就是在?後面打上$filter

/redfish/v1/EventService/Subscriptions/SSE?$filter=RegistryPrefix eq OpenBMC

建立連線後我們可以在Subscriptions中看到RegistryPrefixes裡面有新增OpenBMC了

#/redfish/v1/EventService/Subscriptions/2225775524
{
    "@odata.id": "/redfish/v1/EventService/Subscriptions/2225775524",
    "@odata.type": "#EventDestination.v1_7_0.EventDestination",
    "Context": "",
    "DeliveryRetryPolicy": "TerminateAfterRetries",
    "Destination": "",
    "EventFormatType": "Event",
    "HttpHeaders": [],
    "Id": "2225775524",
    "MessageIds": [],
    "MetricReportDefinitions": [],
    "Name": "Event Destination 2225775524",
    "Protocol": "Redfish",
    "RegistryPrefixes": [
        "OpenBMC"
    ],
    "ResourceTypes": [],
    "SubscriptionType": "SSE"
}

目前OpenBMC支援的SubscriptionType只有SSE和RedfishEvent，目前也在計畫中新增其他SubscriptionType，有興趣的可以參考官方的介紹影片

https://www.youtube.com/watch?v=_8ZnMVcMnbs

采采的生活隨筆

2021年6月27日星期日

[OpenBMC] 從程式碼談 Redfish 的Log 機制 (LogService & EventService)

LogService

Journal - Systemd日誌

Message Registry

EventService

RedfishEvent

SSE(Server-Sent Events)

沒有留言:

張貼留言

2021年6月27日 星期日

[OpenBMC] 從程式碼談 Redfish 的Log 機制 (LogService & EventService)

LogService

Journal - Systemd日誌

Message Registry

EventService

RedfishEvent

SSE(Server-Sent Events)

沒有留言:

張貼留言

2021年6月27日星期日