Re: [PATCH v2 15/20] nvme: add support for scatter gather lists

[Beata Michalska]

On Mon, 25 Nov 2019 at 06:21, Klaus Birkelund <address@hidden> wrote:
>
> On Tue, Nov 12, 2019 at 03:25:18PM +0000, Beata Michalska wrote:
> > Hi Klaus,
> >
> > On Tue, 15 Oct 2019 at 11:57, Klaus Jensen <address@hidden> wrote:
> > > +static uint16_t nvme_map_sgl(NvmeCtrl *n, QEMUSGList *qsg,
> > > +    NvmeSglDescriptor sgl, uint32_t len, NvmeRequest *req)
> > > +{
> > > +    const int MAX_NSGLD = 256;
> > > +
> > > +    NvmeSglDescriptor segment[MAX_NSGLD];
> > > +    uint64_t nsgld;
> > > +    uint16_t status;
> > > +    bool sgl_in_cmb = false;
> > > +    hwaddr addr = le64_to_cpu(sgl.addr);
> > > +
> > > +    trace_nvme_map_sgl(req->cid, NVME_SGL_TYPE(sgl.type), req->nlb, len);
> > > +
> > > +    pci_dma_sglist_init(qsg, &n->parent_obj, 1);
> > > +
> > > +    /*
> > > +     * If the entire transfer can be described with a single data block 
> > > it can
> > > +     * be mapped directly.
> > > +     */
> > > +    if (NVME_SGL_TYPE(sgl.type) == SGL_DESCR_TYPE_DATA_BLOCK) {
> > > +        status = nvme_map_sgl_data(n, qsg, &sgl, 1, &len, req);
> > > +        if (status) {
> > > +            goto unmap;
> > > +        }
> > > +
> > > +        goto out;
> > > +    }
> > > +
> > > +    /*
> > > +     * If the segment is located in the CMB, the submission queue of the
> > > +     * request must also reside there.
> > > +     */
> > > +    if (nvme_addr_is_cmb(n, addr)) {
> > > +        if (!nvme_addr_is_cmb(n, req->sq->dma_addr)) {
> > > +            return NVME_INVALID_USE_OF_CMB | NVME_DNR;
> > > +        }
> > > +
> > > +        sgl_in_cmb = true;
> > > +    }
> > > +
> > > +    while (NVME_SGL_TYPE(sgl.type) == SGL_DESCR_TYPE_SEGMENT) {
> > > +        bool addr_is_cmb;
> > > +
> > > +        nsgld = le64_to_cpu(sgl.len) / sizeof(NvmeSglDescriptor);
> > > +
> > > +        /* read the segment in chunks of 256 descriptors (4k) */
> > > +        while (nsgld > MAX_NSGLD) {
> > > +            nvme_addr_read(n, addr, segment, sizeof(segment));
> > Is there any chance this will go outside the CMB?
> >
>
> Yes, there certainly was a chance of that. This has been fixed in a
> general way for both nvme_map_sgl and nvme_map_sgl_data.
>
> > > +
> > > +            status = nvme_map_sgl_data(n, qsg, segment, MAX_NSGLD, &len, 
> > > req);
> > > +            if (status) {
> > > +                goto unmap;
> > > +            }
> > > +
> > > +            nsgld -= MAX_NSGLD;
> > > +            addr += MAX_NSGLD * sizeof(NvmeSglDescriptor);
> > > +        }
> > > +
> > > +        nvme_addr_read(n, addr, segment, nsgld * 
> > > sizeof(NvmeSglDescriptor));
> > > +
> > > +        sgl = segment[nsgld - 1];
> > > +        addr = le64_to_cpu(sgl.addr);
> > > +
> > > +        /* an SGL is allowed to end with a Data Block in a regular 
> > > Segment */
> > > +        if (NVME_SGL_TYPE(sgl.type) == SGL_DESCR_TYPE_DATA_BLOCK) {
> > > +            status = nvme_map_sgl_data(n, qsg, segment, nsgld, &len, 
> > > req);
> > > +            if (status) {
> > > +                goto unmap;
> > > +            }
> > > +
> > > +            goto out;
> > > +        }
> > > +
> > > +        /* do not map last descriptor */
> > > +        status = nvme_map_sgl_data(n, qsg, segment, nsgld - 1, &len, 
> > > req);
> > > +        if (status) {
> > > +            goto unmap;
> > > +        }
> > > +
> > > +        /*
> > > +         * If the next segment is in the CMB, make sure that the sgl was
> > > +         * already located there.
> > > +         */
> > > +        addr_is_cmb = nvme_addr_is_cmb(n, addr);
> > > +        if ((sgl_in_cmb && !addr_is_cmb) || (!sgl_in_cmb && 
> > > addr_is_cmb)) {
> > > +            status = NVME_INVALID_USE_OF_CMB | NVME_DNR;
> > > +            goto unmap;
> > > +        }
> > > +    }
> > > +
> > > +    /*
> > > +     * If the segment did not end with a Data Block or a Segment 
> > > descriptor, it
> > > +     * must be a Last Segment descriptor.
> > > +     */
> > > +    if (NVME_SGL_TYPE(sgl.type) != SGL_DESCR_TYPE_LAST_SEGMENT) {
> > > +        trace_nvme_err_invalid_sgl_descriptor(req->cid,
> > > +            NVME_SGL_TYPE(sgl.type));
> > > +        return NVME_SGL_DESCRIPTOR_TYPE_INVALID | NVME_DNR;
> > Shouldn't we handle a case here that requires calling unmap ?
>
> Woops. Fixed.
>
> > > +static uint16_t nvme_dma_read_sgl(NvmeCtrl *n, uint8_t *ptr, uint32_t 
> > > len,
> > > +    NvmeSglDescriptor sgl, NvmeCmd *cmd, NvmeRequest *req)
> > > +{
> > > +    QEMUSGList qsg;
> > > +    uint16_t err = NVME_SUCCESS;
> > > +
> > Very minor: Mixing convention: status vs error
> >
>
> Fixed by proxy in another refactor.
>
> > >
> > > +#define NVME_CMD_FLAGS_FUSE(flags) (flags & 0x3)
> > > +#define NVME_CMD_FLAGS_PSDT(flags) ((flags >> 6) & 0x3)
> > Minor: This one is slightly misleading - as per the naming and it's usage:
> > the PSDT is a field name and as such does not imply using SGLs
> > and it is being used to verify if given command is actually using
> > SGLs.
> >
>
> Ah, is this because I do
>
>   if (NVME_CMD_FLAGS_PSDT(cmd->flags)) {
>
> in the code? That is, just checks for it not being zero? The value of
> the PRP or SGL for Data Transfer (PSDT) field *does* specify if the
> command uses SGLs or not. 0x0: PRPs, 0x1 SGL for data, 0x10: SGLs for
> both data and metadata. Would you prefer the condition was more
> explicit?
>
Yeah, it is just not obvious( at least to me)  without referencing the spec
that non-zero value implies SGL usage. Guess a comment would be helpful
but that is not major.

Thanks,

BR
Beata
>
> Thanks!
> Klaus